Multi-Modal Foundation · AIRI-Institute
GENA-LM BERT base (T2T)
A BERT-base genomic foundation model trained on T2T human assemblies for masked DNA sequence understanding.
What GENA-LM BERT base (T2T) is
GENA-LM BERT base (T2T) is a genomic foundation model developed by the AIRI Institute for analysing human DNA sequences. It adapts the standard BERT-base architecture to DNA sequences using masked language modelling, offering a foundation for genomic sequence analysis and classification tasks.
Conventional nucleotide models such as DNABERT rely on k-mer tokenisation, have context windows of 512 nucleotides, and were trained on older assemblies such as GRCh38.p13. GENA-LM addresses this by using byte-pair encoding (BPE) to encompass roughly 4,500 nucleotides within 512 tokens, and was pre-trained on the Telomere-to-Telomere (T2T) human genome assembly enriched with variant data.
How it works
The architecture modifies standard Transformers by applying Pre-Layer normalisation without a final normalisation layer.
Pre-training was conducted for 2,100,000 iterations with a batch size of 256 on the T2T human genome assembly (GCA_009914755.3). Following the masking strategy from BigBird, 15% of tokens are masked and predicted during training. Sequence variation was introduced by sampling mutations from gnomAD 1000 Genomes single-nucleotide polymorphisms.
Key facts
- Architecture
- 12-layer Transformer encoder (Pre-Layer normalisation, BERT-base style)
- Hidden size
- 768
- Attention heads
- 12
- Context length
- 512 tokens (approximately 4,500 nucleotides)
- Pre-training data
- T2T human genome assembly (GCA_009914755.3) with gnomAD SNPs
What you can use it for
- Masked language modelling
- Recover masked bases within DNA sequences up to roughly 4,500 nucleotides in length.
- Promoter region prediction
- Fine-tune the encoder backbone to detect and classify genomic promoter sequences.
- Splice site identification
- Identify splicing donor and acceptor positions using specialised downstream configurations.
- Chromatin and expression modelling
- Map epigenetic features and gene expression using weights adapted on Enformer datasets.
- Sequence annotation
- Score regulatory elements using fine-tuned models configured for DeepSEA and DeepSTARR tasks.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
GENA-LM BERT base (T2T)
Strengths and limits
Good at
- Covers roughly 4,500 nucleotides within a 512-token context window using byte-pair encoding instead of k-mers.
- Trained on the complete Telomere-to-Telomere (T2T) human genome assembly rather than older fragmented assemblies.
- Augmented during training with human genetic diversity sampled from gnomAD single-nucleotide polymorphisms.
- Maintains standard BERT-base dimensions, allowing integration with existing sequence classification workflows.
Watch out for
- Sequence window is constrained to 512 BPE tokens, restricting single-pass analysis of larger genomic regions.
- Pre-training is human-specific, meaning cross-species transfer requires further evaluation or adaptation.
- Loading the model through standard interfaces requires enabling remote code execution or pulling custom modeling scripts.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use GENA-LM BERT base (T2T) commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run GENA-LM BERT base (T2T) without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
How does tokenisation differ between GENA-LM and DNABERT?
GENA-LM uses byte-pair encoding with a 32k vocabulary, allowing a 512-token context to span about 4,500 nucleotides, whereas DNABERT uses k-mers spanning 512 nucleotides.
Which genome reference was used during pre-training?
It was pre-trained on the T2T human genome assembly (GCA_009914755.3), with additional mutation sampling using gnomAD 1000 Genomes SNPs.
Can the model be used for classification downstream tasks?
Yes, sequence classification heads can be attached to the encoder using the custom classification classes provided in the GENA-LM repository.
What fine-tuned checkpoints are available?
Dedicated branches exist for promoter prediction, splice site detection, Enformer-based epigenetic and expression profiling, DeepSEA, and DeepSTARR annotations.
How to cite it
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., Burtsev, M. (2025). GENA-LM: a family of open-source foundational DNA language models for long sequences. Nucleic Acids Research, 53(2), gkae1310. https://doi.org/10.1093/nar/gkae1310
Written from GENA-LM BERT base (T2T)'s own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.