Multi-Modal Foundation · tattabio
gLM2 650M
A 650M-parameter genomic foundation model encoding contigs with both amino acid and nucleotide tokens.
What gLM2 650M is
gLM2 650M is a mixed-modality genomic language model developed by Tatta Bio. It addresses the challenge of sequence-level biological reasoning across genomic contigs by jointly representing both coding and non-coding regions in a unified sequence format.
The model represents genomic scaffolds by encoding protein coding sequences as per-amino-acid tokens and inter-genic sequences as per-nucleotide tokens. Special strand tokens designate positive and negative orientations along the contig, allowing researchers to extract representations that integrate both protein-level and nucleotide-level genomic context.
How it works
gLM2 is built as a transformer encoder trained using a masked language modelling objective with a 30% masking rate. To construct a sequence, protein coding sequences (CDS) are tokenised per amino acid in uppercase, while inter-genic sequences (IGS) are tokenised per nucleotide in lowercase. Each element is prepended with either a <+> or <-> token to denote its strand orientation.
The 650M parameter model was pretrained on 315 billion tokens from the Open MetaGenomic (OMG) corpus using bfloat16 mixed-precision and the AdamW optimiser. Training data was deduplicated using SemDedup at an embedding distance threshold of 2e-3, which pruned 49% of the initial dataset to reduce near-duplicate sequences.
Key facts
- Model architecture
- Transformer encoder
- Parameter count
- 650M
- Context length
- 4096 tokens
- Training data
- OMG (Open MetaGenomic) dataset
- Pretraining tokens
- 315B
- Training objective
- Masked language modelling (30% masking rate)
- Required GPU memory
- 16 GB
What you can use it for
- Genomic sequence embedding extraction
- Generate hidden-state sequence representations for entire contigs comprising both coding genes and inter-genic regions.
- Masked sequence prediction
- Predict missing or masked amino acids and nucleotides within mixed genomic scaffolds using its fill-mask pipeline.
- Sequence-level biological reasoning
- Analyse multi-element genomic contigs while accounting for reading direction and strand orientation.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
gLM2 650M
Strengths and limits
Good at
- Encodes both amino acid coding sequences and nucleotide inter-genic sequences in a single mixed-modality model.
- Maintains genomic orientation using explicit strand-indicator tokens (<+> and <->).
- Trained on a semantically deduplicated metagenomic dataset with 49% of near-duplicate data removed.
- Offers a 4096-token context window with single-nucleotide resolution across non-coding spans.
Watch out for
- Strictly requires exact case conventions, requiring uppercase for amino acids and lowercase for nucleotides to prevent token collisions.
- Requires custom remote code execution when loading via standard model libraries.
- Needs 16 GB of GPU memory to run the inference pipeline.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use gLM2 650M commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run gLM2 650M without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
How does gLM2 distinguish between amino acids and nucleotides in an input sequence?
The tokenizer expects all amino acid characters in protein coding regions to be uppercase, and all nucleotide characters in inter-genic regions to be lowercase.
How are genomic strands represented in the sequence?
Every genomic segment is prepended with a special token, either <+> for the positive strand or <-> for the negative strand.
What is the context length limit of gLM2 650M?
The model has a maximum context length of 4096 tokens.
How to cite it
Cornman, Andre and West-Roberts, Jacob and Camargo, Antonio Pedro and Roux, Simon and Beracochea, Martin and Mirdita, Milot and Ovchinnikov, Sergey and Hwang, Yunha. The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling. bioRxiv, 2024. doi:10.1101/2024.08.14.607850
Written from gLM2 650M's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.