Multi-Modal Foundation · AIRI-Institute

GENA-LM BERT large (T2T)

Genomic foundation model based on BERT large, trained on the T2T human genome assembly for sequence analysis.

What GENA-LM BERT large (T2T) is

GENA-LM BERT large (T2T) is a genomic foundation model developed by the AIRI-Institute to interpret human DNA sequences. It provides contextual nucleotide representations intended to support variant interpretation, regulatory element identification, and diverse sequence-based genomic tasks.

Unlike earlier genomic models that trained on older references such as GRCh38.p13 using short k-mers, this model was trained on the telomere-to-telomere (T2T) human genome assembly. By combining byte pair encoding with a BERT-large transformer architecture, it captures longer stretches of sequence context to assist genetic and clinical researchers.

How it works

It uses byte pair encoding (BPE) rather than overlapping k-mers, allowing its maximum sequence context of 512 tokens to span roughly 4,500 nucleotides.

Pre-training followed a masked language modelling objective where 15% of tokens were masked, matching the BigBird masking approach. The network was trained for 1,750,000 iterations at a batch size of 256 on the T2T human genome assembly (GCA_009914755.3), with sequence diversity augmented by sampling single nucleotide polymorphisms from the gnomAD 1000-genomes dataset.

Key facts

Architecture: Transformer masked language model (BERT-large with Pre-Layer normalisation)
Layers and attention heads: 24 layers, 16 attention heads
Hidden dimension: 1024
Context length: 512 BPE tokens (approximately 4,500 nucleotides)
Training data: T2T human genome assembly (GCA_009914755.3) augmented with gnomAD SNPs
Pre-training iterations: 1,750,000 steps with batch size 256
Hardware requirement: 16 GB GPU memory

What you can use it for

Promoter prediction: Identify promoter regions using sequence classification heads, with established fine-tuning implementations for 300 base-pair and 2,000 base-pair windows.
Splice site identification: Locate splicing junctions along genomic DNA using fine-tuned sequence classification architectures.
Variant interpretation and masked language modelling: Assess mutation likelihoods and evaluate local sequence patterns by predicting masked nucleotide tokens.
Genomic sequence classification: Fine-tune the pre-trained contextual embeddings with classification heads to predict regulatory function and sequence annotations.

Run it here

No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.

Runs on On-demand GPU

Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.

Run this model

GENA-LM BERT large (T2T)

Strengths and limits

Good at

Pre-trained on the telomere-to-telomere human genome assembly rather than older, incomplete reference genomes.
Uses byte pair encoding tokenisation to cover roughly 4,500 nucleotides within a 512-token context, substantially longer than k-mer-based models of equal token length.
Incorporates human genetic diversity by augmenting pre-training sequences with gnomAD single nucleotide polymorphisms.

Watch out for

Sequence input is capped at 512 BPE tokens, meaning longer genomic loci require sequence chunking.
Requires 16 GB of GPU memory for standard pipeline execution.
Requires fine-tuning or task-specific classification heads to perform supervised tasks such as splice site or promoter detection.

Inputs and outputs

You provide

Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.

You get back

Labels with confidence scores, highest first.

Licence

Check the licence before commercial use · See model card

The See model card licence carries terms worth reading before you depend on it.

Questions

Can I use GENA-LM BERT large (T2T) commercially?

Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.

How do I run GENA-LM BERT large (T2T) without installing anything?

Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.

How does this model differ from DNABERT?

It uses byte pair encoding (BPE) instead of k-mers, extends sequence coverage to around 4,500 nucleotides across 512 tokens instead of 512 nucleotides, and is pre-trained on the T2T human genome assembly rather than GRCh38.p13.

What training data was used for pre-training?

The model was pre-trained on the T2T human genome assembly (NCBI GCA_009914755.3) augmented with human variants sampled from the gnomAD dataset.

Can this model be fine-tuned for sequence classification?

Yes, it can be paired with classification heads such as BertForSequenceClassification from the GENA-LM repository or Hugging Face AutoModel.

How to cite it

Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2025). GENA-LM: a family of open-source foundational DNA language models for long sequences. Nucleic Acids Research, 53(2), gkae1310. https://doi.org/10.1093/nar/gkae1310

Written from GENA-LM BERT large (T2T)'s own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.