Metagenomics · LiteFold
Mgnify
A multi-terabyte index and distribution of microbiome protein sequences and tables from EMBL-EBI MGnify.
What Mgnify is
Mgnify (LiteFold/Mgnify) is a distribution of protein sequence catalogues and relational metadata derived from the EMBL-EBI MGnify platform. The underlying database comprises over 2.4 billion non-redundant protein sequences predicted from metagenomic, metatranscriptomic, and metabarcoding assemblies across diverse biomes, complete with taxonomic and functional annotations.
Metagenomic sequence databases often present engineering challenges due to their massive size and multi-terabyte payloads. This distribution provides structured access via a Parquet-based index file, chunked compressed FASTA shards, and partitioned JSONL metadata tables, addressing the challenge of navigating and selectively downloading massive environmental protein datasets.
In structural biology and machine learning, metagenomic proteins are vital for constructing multiple sequence alignments (MSAs) for deep protein families under-represented in standard databases like UniProt. This resource facilitates large-scale sequence retrieval and machine learning workflows for protein structure prediction, sequence analysis, and functional annotation.
How it was made
The dataset organizes MGnify sequence and metadata files into a Parquet index that tracks 3,226 repository files across 34 metadata attributes, preventing unintended multi-terabyte full-repository downloads.
The default index provides a deterministic train/test partition using a hash-based split (sha256 of the file identifier modulo 10) into 2,902 training entries and 324 testing entries. Users can query the lightweight Parquet index to retrieve specific shard paths, glob patterns, and file metadata before selectively streaming or downloading raw FASTA sequences or JSONL metadata tables.
Key facts
- Underlying sequences
- Over 2.4 billion non-redundant sequences
- Sequence shards
- 3,148 files (342,815,994,580 bytes)
- Sequence format
- Zstandard-compressed FASTA (.fasta.zst)
- Table format
- JSON Lines (.jsonl)
- Index format
- Apache Parquet (.parquet)
- Index rows
- 3,226 rows (2,902 train / 324 test)
- Logical tables
- 28 tables (2,380,980,172,257 bytes)
- Original provider
- EMBL-EBI MGnify
What you can use it for
- Multiple sequence alignment for structure prediction
- Supplies deep metagenomic sequence diversity alongside UniRef and BFD to construct multiple sequence alignments (MSAs) for AlphaFold2 and related protein structure prediction architectures.
- Metagenomic protein function annotation
- Enables training and benchmarking of deep learning models on environmental protein sequences linked to Pfam and metadata annotations.
- Genomic context and assembly exploration
- Allows researchers to trace environmental proteins back to their source metagenomic assemblies, biomes, and sample metadata using the relational JSONL tables.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from LiteFold/Mgnify at the moment you asked. Nothing is cached or stored — every row above came from that request.
How the columns are distributed
Over 2,902 rows of default/train
access_note
string_label- Default config indexes Mgnify files. Stream raw FASTA/table payloads from sequences/ and tables/ with huggingface_hub.100%
compression
string_label- zstd97%
- jsonl2%
- json0%
dataset_id
string_label- mgnify_proteins100%
download_pattern
string_label- sequences/sequence_mgnify_current_release_mgy_clusters.fa.gz/shard-*.fasta.zst22%
- sequences/sequence_mgnify_current_release_mgy_proteins_23.fa.gz/shard-*.fasta.zst3%
- sequences/sequence_mgnify_current_release_mgy_proteins_22.fa.gz/shard-*.fasta.zst3%
file_id
string_text- min
- 9
- median
- 85
- mean
- 84.97
- max
- 98
is_original_table_copy
bool- False100%
- True0%
is_sequence_shard
bool- True97%
- False3%
is_split_manifest
bool- False100%
- True0%
is_split_part
bool- False99%
- True1%
is_table_file
bool- False98%
- True2%
logical_table_bytes_total
int- min
- 2381.0B
- median
- 2381.0B
- mean
- 2381.0B
- max
- 2381.0B
logical_table_count_total
int- min
- 28
- median
- 28
- mean
- 28
- max
- 28
logical_table_size_bytes
int- min
- -1
- median
- -1
- mean
- 8.3B
- max
- 709.8B
part_index
int- min
- -1
- median
- -1
- mean
- -0.90
- max
- 17
path
string_text- min
- 9
- median
- 85
- mean
- 84.97
- max
- 98
repo_file_count
int- min
- 3,226
- median
- 3,226
- mean
- 3,226
- max
- 3,226
repo_id
string_label- LiteFold/Mgnify100%
repo_total_bytes
int- min
- 3156.1B
- median
- 3156.1B
- mean
- 3156.1B
- max
- 3156.1B
role
string_label- sequence_shard97%
- table_split_part1%
- table_jsonl1%
sequence_shard_bytes_total
int- min
- 342.8B
- median
- 342.8B
- mean
- 342.8B
- max
- 342.8B
sequence_shard_count_total
int- min
- 3,148
- median
- 3,148
- mean
- 3,148
- max
- 3,148
sequence_source_bytes
int- min
- -1
- median
- 10.9B
- mean
- 25.0B
- max
- 75.1B
sequence_source_shard_count
int- min
- -1
- median
- 99
- mean
- 234.57
- max
- 717
shard_index
int- min
- -1
- median
- 60
- mean
- 118.45
- max
- 717
size_bytes
int- min
- 467
- median
- 110M
- mean
- 1.0B
- max
- 193.2B
source_family
string_label- mgy_clusters23%
- mgy_proteins_233%
- mgy_proteins_223%
source_file
string_label- sequence/mgnify/current_release/mgy_clusters.fa.gz22%
- sequence/mgnify/current_release/mgy_proteins_23.fa.gz3%
- sequence/mgnify/current_release/mgy_proteins_22.fa.gz3%
source_sha
string_label- 4823aa306c32be5800a6ad6a1a6afd256bece606100%
source_slug
string_label- sequence_mgnify_current_release_mgy_clusters.fa.gz22%
- sequence_mgnify_current_release_mgy_proteins_22.fa.gz3%
- sequence_mgnify_current_release_mgy_proteins_23.fa.gz3%
split_bucket
int- min
- 1
- median
- 5
- mean
- 5.02
- max
- 9
split_chunk_bytes
int- min
- -1
- median
- -1
- mean
- 689M
- max
- 40.0B
split_part_count
int- min
- -1
- median
- -1
- mean
- -0.78
- max
- 18
table_repo_bytes_total
int- min
- 2813.3B
- median
- 2813.3B
- mean
- 2813.3B
- max
- 2813.3B
table_repo_file_count_total
int- min
- 76
- median
- 76
- mean
- 76
- max
- 76
Strengths and limits
Good at
- Contains over 2.4 billion non-redundant predicted protein sequences covering a wide spectrum of environmental biomes.
- Organised with a lightweight Parquet index file that enables targeted downloading or streaming without requiring complete multi-terabyte repository clones.
- Maintains relational links between protein sequences, Pfam functional annotations, and source sample assemblies.
- FASTA sequences are pre-sharded and compressed with Zstandard for streamable, random access to sequence subsets.
Watch out for
- The provided train/test splits reflect repository file partitions rather than biologically distinct sequence-level or cluster-level splits.
- The full raw payload exceeds 3 TB, requiring significant local storage if downloaded in bulk.
- Raw table files are multi-terabyte JSONL dumps that are not natively loaded via standard Hugging Face datasets configs without custom filtering.
Questions
Can I look inside Mgnify without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is Mgnify under?
Its authors declare cc-by-4.0. Read the terms on the original page before reusing the data.
Can I use the default train and test splits directly for training a sequence model?
No. The default splits are file-index partitions based on file identifier hashes. For biological model training, sequence-level or cluster-level splits must be generated after downloading the relevant data.
What is the primary role of MGnify sequences in protein structure prediction?
In structure prediction systems like AlphaFold2, MGnify serves as the metagenomic database component in MSA search pipelines, enriching poorly represented protein families with diverse environmental sequence homologues.
How can I access individual FASTA sequences without downloading hundreds of gigabytes?
You can stream individual sequence shards directly over HTTP using the Hugging Face filesystem interface and the Python zstandard library without cloning the entire sequence catalogue.
What formats are used for sequence and metadata files?
The primary navigation index is provided in Parquet format, sequences are stored as Zstandard-compressed FASTA shards (.fasta.zst), and metadata tables are stored as plain or partitioned JSON Lines (.jsonl).
How to cite it
Richardson, Lorna and Allen, Ben and Baldi, Germana and Beracochea, Martin and Bileschi, Maxwell L. and Burdett, Tony and Burgin, Josephine and Caballero-Pérez, Juan and Cochrane, Guy and Colwell, Lucy J. and Curtis, Tom and Escobar-Zepeda, Alejandra and Gurbich, Tatiana A. and Kale, Varsha and Korobeynikov, Anton and Raj, Shriya and Rogers, Alexander B. and Sakharova, Ekaterina and Sanchez, Santiago and Wilkinson, Darren J. and Finn, Robert D. MGnify: the microbiome sequence data analysis resource in 2023. Nucleic Acids Research, 51(D1):D753–D759, 2023. DOI: 10.1093/nar/gkac1080.
Written from Mgnify's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.