Microbial Genomics · AllTheBacteria
Bac-Corpus-protein-sequences-high-diversity
A corpus of approximately 940 million clustered bacterial protein sequences for pretraining language models.
What Bac-Corpus-protein-sequences-high-diversity is
BacCorpus Protein 90 is a large-scale collection of representative bacterial protein sequences released by AllTheBacteria. The dataset contains approximately 940 million protein sequences clustered at 90% sequence identity and 90% sequence coverage, designed to maximise sequence diversity for computational biology applications.
Bacterial protein sequences encapsulate substantial conserved functional signals across bacterial genomes, supporting the computational modelling of gene function, protein composition, and whole-genome phenotypes. This dataset provides a non-redundant training resource aimed at pretraining bacterial protein language models and genomic language models.
The data originates from BacCorpus, a bacterial genome resource compiled from millions of quality-controlled isolate genomes and metagenome-assembled genomes gathered across multiple major public microbial repositories.
How it was made
The corpus was created by aggregating bacterial genome assemblies from six public repositories: MGnify, SPIRE, HRGM, GTDB, mOTUs DB, and AllTheBacteria. Genomes underwent uniform processing and annotation to identify and extract predicted protein-coding sequences.
To eliminate extreme sequence redundancy while preserving functional diversity, the extracted protein sequences were clustered using MMSeqs2 (version 18.8cc5c) at thresholds of 90% sequence identity and 90% coverage. The representative sequences from each cluster were then exported into Parquet format.
Key facts
- Scale
- Approximately 940 million representative sequences
- Data type
- Bacterial protein sequences
- Clustering threshold
- 90% sequence identity and 90% coverage
- Clustering tool
- MMSeqs2 (version 18.8cc5c)
- Format
- Parquet
- Source databases
- MGnify, SPIRE, HRGM, GTDB, mOTUs DB, AllTheBacteria
What you can use it for
- Protein language model pretraining
- Pretraining transformer and sequence models on hundreds of millions of diverse bacterial amino acid sequences.
- Genomic language model training
- Supplying representative protein-coding features to models that learn bacterial gene functions and genome-scale phenotypes.
- Diversity and redundancy research
- Studying the breadth of bacterial protein sequence diversity and testing sequence deduplication strategies in microbiology.
- Representative protein sampling
- Subsampling diverse sets of bacterial proteins for specialised downstream machine learning tasks.
Explore the data
No licence declared
The authors have not published a licence, which means no permission to redistribute the data has been granted to anyone. The rows are not served here; read them on the original dataset page.Strengths and limits
Good at
- Drawn from millions of isolate genomes and metagenome-assembled genomes across multiple public microbial databases.
- Clustering with MMSeqs2 at 90% identity and coverage substantially decreases sequence redundancy while retaining biological diversity.
- Contains approximately 940 million representative sequences formatted in Parquet for structured data loading and streaming.
Watch out for
- The corpus is limited strictly to bacterial proteins and does not include archaeal or eukaryotic sequences.
- Because of the dataset's substantial volume, local disk downloads are impractical and streaming workflows are recommended.
Questions
Can I look inside Bac-Corpus-protein-sequences-high-diversity without downloading it?
Not here: it declares no licence, so no permission to redistribute its rows has been granted. Its description and statistics are still shown.
What licence is Bac-Corpus-protein-sequences-high-diversity under?
Its authors have not declared a licence, which means no permission to reuse or redistribute it has been granted.
What organisms are covered by this dataset?
The dataset exclusively covers bacteria, incorporating both isolate genomes and metagenome-assembled genomes.
Which databases provided the source genomes?
Genomes were gathered from MGnify, SPIRE, HRGM, GTDB, mOTUs DB, and AllTheBacteria.
How was sequence clustering conducted?
Predicted protein-coding sequences were clustered using MMSeqs2 (version 18.8cc5c) applying cut-offs of 90% sequence identity and 90% coverage.
How should this dataset be loaded for model training?
Owing to its scale of roughly 940 million sequences, streaming the dataset using data loading libraries is recommended rather than downloading it in full to local storage.
Written from Bac-Corpus-protein-sequences-high-diversity's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.