Genomics · InstaDeepAI
multi_species_genomes
A 174-billion-nucleotide multi-species genomic dataset for pre-training DNA foundation models across diverse taxa.
What multi_species_genomes is
multi_species_genomes is a large-scale collection of genomic DNA sequences compiled by InstaDeepAI to serve as the pre-training corpus for the Nucleotide Transformer foundation model family. The dataset contains 174 billion nucleotides derived from 850 distinct species, corresponding to approximately 29 billion tokens.
The dataset was built by harvesting genomes from the National Center for Biotechnology Information (NCBI) database. The curators downsampled the collection by selecting a single species per genus while deliberately retaining several genomes that are extensively studied in scientific literature.
The taxonomic coverage spans bacteria, fungi, invertebrates, protozoa, mammalian vertebrates, and other vertebrates. Plant and viral genomes were excluded because their regulatory elements diverge from the functional genomics tasks examined by the developers.
How it was made
Whole-genome assemblies from NCBI RefSeq FASTA files were split into contiguous segments of either 6,200 or 12,200 base pairs, depending on configuration. When iterated sequentially without shuffling, the first 100 nucleotides of a sequence duplicate the final 100 base pairs of the preceding sequence, and the final 100 nucleotides match the start of the next sequence.
This overlapping configuration allows training routines to pick a random nucleotide within the initial 200 base pairs to initiate tokenisation. This technique ensures complete chromosomal coverage while exposing models to alternative token segmentations of the same sequence across different training epochs. Each record provides the DNA sequence, an organism description with NCBI identifier, the sequence start and end indices, and the source FASTA URL. The dataset is organised into train, validation, and test splits.
Key facts
- Total nucleotides
- 174 billion (~29 billion tokens)
- Total species
- 850 species
- Sequence lengths
- 6,200 or 12,200 base pairs
- Taxonomic breakdown
- 667 bacteria, 57 other vertebrates, 46 fungi, 39 invertebrates, 31 mammalian vertebrates, 10 protozoa
- Excluded taxa
- Plants and viruses
- Data fields
- sequence, description, start_pos, end_pos, fasta_url
- Data splits
- Train, validation, test
- Primary developer
- InstaDeepAI
What you can use it for
- Genomic foundation model pre-training
- Training large autoregressive or masked language models on multi-taxa DNA to capture universal sequence motifs and biological representations.
- Cross-species comparative sequence analysis
- Benchmarking tokenisation strategies and evolutionary conservation across diverse non-plant organismal groups.
- Dynamic token offset training
- Using the embedded 100-nucleotide sequence overlap to generate variable token boundary alignments across training epochs without losing chromosomal coverage.
Explore the data
No licence declared
The authors have not published a licence, which means no permission to redistribute the data has been granted to anyone. The rows are not served here; read them on the original dataset page.Strengths and limits
Good at
- Spans 850 diverse species across six major biological classes, representing 174 billion base pairs.
- Engineered with 100-base-pair boundary overlaps that facilitate data augmentation via randomised tokenisation offsets.
- Preserves genomic coordinates and direct source URLs back to NCBI RefSeq FASTA files for every sequence window.
Watch out for
- Completely excludes viral and plant sequences due to differences in regulatory machinery.
- Restricted to a single representative per genus alongside standard reference organisms, omitting fine-grained intra-genus variation.
- Contains raw nucleotide strings without accompanying gene annotations, variant tags, or functional labels.
Questions
Can I look inside multi_species_genomes without downloading it?
Not here: it declares no licence, so no permission to redistribute its rows has been granted. Its description and statistics are still shown.
What licence is multi_species_genomes under?
Its authors have not declared a licence, which means no permission to reuse or redistribute it has been granted.
Which biological kingdoms and classes are represented?
The dataset contains 667 bacterial species, 57 non-mammalian vertebrates, 46 fungi, 39 invertebrates, 31 mammalian vertebrates, and 10 protozoan species.
Why are plants and viruses absent from the dataset?
Plants and viruses were intentionally excluded during curation because their regulatory elements differ substantially from those targeted in the associated downstream genomics tasks.
How long is each sequence window?
Depending on the dataset configuration used, individual sequence records are either 6,200 or 12,200 base pairs long.
How does the sequence overlap support model training?
Consecutive unshuffled sequences share 100-nucleotide overlaps at their boundaries. This allows tokenisation to start from a randomly chosen position within the first 200 base pairs, exposing models to different token splits across epochs while retaining total sequence coverage.
How to cite it
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G. et al. (2023). The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv, 2023.01.11.523679v1.
Written from multi_species_genomes's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.