Genomic Pretraining Corpus · HuggingFaceBio
carbon-pretraining-corpus
A 1.1-trillion-nucleotide DNA and RNA corpus spanning eukaryotes and prokaryotes for genomic foundation models.
What carbon-pretraining-corpus is
The Carbon pretraining corpus is a curated genomic dataset of 173 million DNA and RNA sequences comprising approximately 1.1 trillion base pairs (180 billion tokens using a 6-mer tokenizer). It was assembled by HuggingFaceBio to train the Carbon family of genomic foundation models (Carbon-500M, Carbon-3B, and Carbon-8B), capturing biological sequence diversity across both eukaryotic and prokaryotic life.
Genomic sequence modelling requires diverse training data to capture evolutionary patterns, gene structures, and regulatory contexts. This dataset addresses that requirement by unifying eukaryotic chromosomal contigs with curated gene-level annotations, dense prokaryotic genomes, and messenger RNA transcripts with and without splice-junction augmentations.
The corpus is organised into four primary configurations alongside a pre-sampled 10-billion-token eukaryotic subset designed for faster, smaller-scale pretraining runs. It supports language modelling on genomic sequences by structuring continuous DNA sequences into clean, format-standardised parquet tables.
How it was made
The corpus is compiled from upstream biological data repositories, primarily GenerTeam's GENERator eukaryotic pretraining dataset and Arc Institute's OpenGenome2. The eukaryotic portion incorporates genomes across fungi, plants, protozoa, invertebrates, and vertebrates, restricted to sequences of 100 kilobase pairs or fewer. The prokaryotic portion integrates bacterial and archaeal genomes sourced from GTDB v220 and IMG/PR. The messenger RNA portions provide spliced transcript sequences, with the augmented split adding 1,024 base pairs of upstream promoter sequence and 32 base pairs of exon-flanking sequence linked by a special delimiter token.
For model training, sequences are divided into discrete units, such as non-overlapping 6-mer windows (one token per six base pairs). In the eukaryotic split, each entry provides metadata including taxonomy, species type, gene type, strand, and chromosomal coordinates. These metadata fields can be prepended as conditioning tags to a sequence at tokenisation time or omitted via random dropout to support unconditional as well as conditional generation.
Key facts
- Token volume
- 180 billion tokens (with a 6-mer tokenizer)
- Data format
- Parquet
- Eukaryote sequence limit
- 100 kilobase pairs (kbp)
- Included domains
- Eukaryota, Bacteria, Archaea, and mRNA transcripts
What you can use it for
- Genomic Foundation Model Pretraining
- Pretraining autoregressive or masked sequence models across cross-species DNA and RNA sequence distributions.
- Eukaryotic Gene Context Modelling
- Training sequence models to identify regulatory regions, coding sequences, and taxonomy-specific genomic features using structured sequence metadata.
- Splice Site and Promoter Sequence Analysis
- Studying transcriptional signals using augmented mRNA sequences containing promoter flanks and exon splice junctions.
- Prokaryotic Genome Modelling
- Evaluating language model perplexity and representation learning on compact bacterial and archaeal chromosomal sequences from GTDB and IMG/PR.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from HuggingFaceBio/carbon-pretraining-corpus at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- Spans multiple biological domains, combining eukaryotic genomes, prokaryotic genomes, and spliced mRNA transcripts.
- Provides rich sequence-level metadata for eukaryotic contigs, including taxonomy, gene biotype, strand, and genomic coordinates.
- Excludes eukaryotic sequences longer than 100 kbp to improve performance on downstream DNA benchmarks.
- Includes a pre-sampled 10-billion-token eukaryotic subset that maintains the natural species distribution for efficient experimentation.
Watch out for
- Eukaryotic sequences exceeding 100 kbp are removed, requiring concatenation of contiguous genes if long chromosomal contexts are needed.
- Metadata conditioning annotations are only present on the eukaryotic split and are absent from the OpenGenome2 prokaryotic and mRNA splits.
Questions
Can I look inside carbon-pretraining-corpus without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is carbon-pretraining-corpus under?
Its authors declare other. Read the terms on the original page before reusing the data.
What is the pretraining data mix used for the Carbon model?
The target mixture consists of approximately 70% eukaryote_generator, 16% mrna_evo2, 10% prokaryote_evo2, and 4% mrna_splice_evo2.
How are sequences tokenised?
The dataset sequences are plain nucleotide strings (A, T, G, C) that can be processed with tokenizers such as Carbon's 6-mer tokenizer, which splits sequences into non-overlapping windows of six bases.
What is the difference between mrna_evo2 and mrna_splice_evo2?
Both subsets cover the same underlying transcripts, but mrna_splice_evo2 appends 1,024 base pairs of upstream promoter sequence and 32 base pairs flanking each exon boundary, joining the chunks with an '@' separator.
What is included in the eukaryote_generator_10B_subset?
It is a 60-Gbp (10-billion-token) uniform row sample of the full eukaryotic split that preserves natural species proportions across vertebrates, invertebrates, plants, fungi, and protozoa.
How to cite it
@article{allal2026carbon, title={Carbon: Decoding the Language of Life}, author={Allal, Loubna Ben and Li, Qiuyi and Fiusco, Maurizio and Tunstall, Lewis and Rasul, Kashif and Beeching, Ed and Aubakirova, Dana and Pati{\~n}o, Carlos and Frere, Thibaud and Lozhkov, Anton and others}, journal={bioRxiv}, pages={2026--05}, year={2026}, publisher={Cold Spring Harbor Laboratory} }
Written from carbon-pretraining-corpus's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.