Demonstration only. Corollary is a non-commercial research demo — no warranty, no support, not for commercial use or clinical decisions. Read the disclaimer

Corollary Labs
All datasets

Protein Sequences · LiteFold

NCBI

A Parquet shard index of NCBI RefSeq curated protein sequences and metadata for biological research.

Explore the dataOriginal source

What NCBI is

LiteFold/NCBI provides a structured Parquet index for the protein subset of the National Center for Biotechnology Information (NCBI) Reference Sequence (RefSeq) collection. RefSeq compiles a curated, non-redundant catalogue of genomic, transcript, and protein sequences representing organisms across the tree of life.

Protein records within this collection are derived from expert curation, the NCBI Eukaryotic and Prokaryotic Genome Annotation pipelines, and data coordinated with collaborating resources. Each protein entry is cross-linked to corresponding source genome and transcript records, with identifier prefixes indicating curation status: NP_ for curated sequences, XP_ for predicted sequences, WP_ for non-redundant prokaryotic proteins, and YP_ for organelle and viral proteins.

The index organizes 459,415,871 protein records comprising 179,203,453,293 amino acid residues from 1,725 source FASTA files. It indexes sequence shards compressed with Zstandard alongside record-level JSONL metadata files, enabling targeted retrieval and analysis of massive sequence stores.

How it was made

The dataset was created using a normalization script (scripts/prepare_ncbi_dataset.py) that indexes 1,725 source FASTA files divided into sequence shards and per-record metadata JSONL files. The sequence shards are stored in Zstandard-compressed FASTA format (.fasta.zst).

Each row in the Parquet index corresponds to a specific file in the repository, recording attributes such as file role (sequence shard, metadata records, or source manifest), path, shard index, file size in bytes, and residue and record counts. Data splits are deterministically assigned by computing the SHA-256 hash of the stable file identifier modulo 10, reserving bucket 0 for testing (502 rows) and buckets 1 to 9 for training (4,676 rows).

Key facts

Total protein records
459,415,871
Total residues
179,203,453,293
Source FASTA files
1,725
Sequence shards
1,725
Index rows
5,178
Train split rows
4,676
Test split rows
502
Compressed shard size
78,108,688,857 bytes
Metadata JSONL size
158,533,041,909 bytes
Index format
Parquet
Sequence file format
Zstandard-compressed FASTA (.fasta.zst)
Metadata format
JSONL

What you can use it for

Protein subset filtering by curation tier
Isolate curated (NP_), predicted (XP_), prokaryotic (WP_), or organelle and viral (YP_) sequences using RefSeq accession prefixes for tailored sequence analyses.
Selective sequence shard extraction
Query the Parquet index to locate, filter, and stream specific sequence shards or source FASTA archives without processing the full sequence repository.
Partitioned model training
Use the deterministic SHA-256-based file splits to train and evaluate protein sequence models on predefined train and test file partitions.
Annotation and provenance tracking
Trace protein entries to their corresponding source FASTA files, aggregate manifests, and source genomes via associated JSONL metadata files.

Explore the data

The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.

Explore

Read the real rows without downloading anything

Reading rows…

LiveRead from LiteFold/NCBI at the moment you asked. Nothing is cached or stored — every row above came from that request.

How the columns are distributed

Over 4,676 rows of default/train

access_note

string_label
  • File/shard index for NCBI RefSeq protein; download sequence shards for FASTA records.100%

compression

string_label
  • zstd33%

66.7% empty

dataset_id

string_label
  • ncbi_refseq_protein100%

download_pattern

string_text
min
9
median
94
mean
93.27
max
104

file_id

string_text
min
9
median
98
mean
94.94
max
109

is_metadata_records

bool
  • False67%
  • True33%

is_sequence_shard

bool
  • False67%
  • True33%

path

string_text
min
9
median
98
mean
94.94
max
109

records_in_source

int
min
1
median
300k
mean
267k
max
800k

0.06% empty

records_total

int
min
459M
median
459M
mean
459M
max
459M

repo_id

string_label
  • LiteFold/NCBI100%

residues_in_source

int
min
165
median
103M
mean
104M
max
335M

0.06% empty

residues_total

int
min
179.2B
median
179.2B
mean
179.2B
max
179.2B

role

string_label
  • source_manifest33%
  • sequence_shard33%
  • metadata_records33%

shard_index

int
min
0
median
0
mean
0
max
0

66.7% empty

shards_in_source

int
min
1
median
1
mean
1
max
1

0.06% empty

size_bytes

int
min
198
median
22M
mean
46M
max
277M

source_file

string_text
min
66
median
75
mean
71.34
max
76

0.06% empty

source_sha

string_label
  • ca26266937eac32428823d12614ba3c639a36ba0100%

source_slug

string_text
min
66
median
75
mean
71.34
max
76

0.06% empty

split_bucket

int
min
1
median
5
mean
4.94
max
9

total_shards

int
min
1,725
median
1,725
mean
1,725
max
1,725

Strengths and limits

Good at

  • Accession prefixes directly indicate sequence provenance and whether a record was expert-curated or computationally predicted.
  • Provides deterministic SHA-256 partition assignments across 5,178 shard and metadata file entries.
  • Combines compact Parquet indexing with Zstandard-compressed FASTA shards and JSONL metadata for granular data retrieval.

Watch out for

  • Contains file paths, manifests, and shard statistics rather than inlining sequence strings directly within the Parquet tables.
  • Extracting sequence records requires external decompression of Zstandard-compressed FASTA shards.
  • Dataset splits are partitioned strictly by file identifier hash rather than by sequence identity, structure, or taxonomic clade.

Questions

Can I look inside NCBI without downloading it?

Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.

What licence is NCBI under?

Its authors declare other. Read the terms on the original page before reusing the data.

What do the accession prefixes in RefSeq protein records signify?

Prefixes indicate curation status: NP_ denotes curated proteins, XP_ denotes computationally predicted proteins, WP_ denotes non-redundant prokaryotic proteins, and YP_ marks organelle and viral proteins.

What data formats are indexed in this repository?

The index is provided in Parquet format, while the underlying files consist of Zstandard-compressed FASTA shards (.fasta.zst) and per-record metadata in JSONL format.

How are the train and test splits created?

Splits are determined using the formula sha256(file_id) % 10. Bucket 0 is allocated to the test split (502 rows), and buckets 1 through 9 form the train split (4,676 rows).

Does the Parquet index table contain the amino acid sequences directly?

No, the Parquet table serves as a shard index containing metadata, record counts, residue counts, and file paths pointing to external sequence shards and JSONL files.

How to cite it

Goldfarb, Tamara, Vamsi K. Kodali, Shashikant Pujar, Vyacheslav Brover, Barbara Robbertse, Catherine M. Farrell, Dong-Ha Oh, et al. 'NCBI RefSeq: reference sequence standards through 25 years of curation and annotation.' Nucleic Acids Research 53, no. D1 (2025): D243–D257. https://doi.org/10.1093/nar/gkae1038

Written from NCBI's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.

Related datasets