Multi-Modal Foundation · AIRI-Institute
ModernGENA large
A 377M-parameter ModernBERT foundation model for vertebrate DNA sequence modelling and masked sequence prediction.
What ModernGENA large is
ModernGENA large is a 377-million-parameter DNA foundation model developed by the AIRI-Institute. It rebuilds the GENA-LM framework using the ModernBERT bidirectional encoder architecture, adapting modern transformer design choices to genomic sequence processing.
The model addresses the computational and architectural demands of genomic sequence representation by serving as an efficient baseline for genomic research. It demonstrates competitive downstream performance on the Nucleotide Transformer benchmark.
How it works
ModernGENA large is based on a modernized BERT-style encoder incorporating rotary position embeddings (RoPE), pre-norm transformer blocks, GeGLU feed-forward networks, and hybrid local/global attention. It employs end-to-end unpadding, FlashAttention compatibility, and support for torch.compile to maximise throughput and support long genomic contexts.
Pre-training was conducted on 443 vertebrate genome assemblies totalling 353,574,093,776 base pairs, encompassing both forward and reverse complement sequences while omitting sequences with ambiguous characters outside A, C, G, and T. To enrich biologically active sequence and suppress simple repeats, pre-training regions were sampled within windows spanning 16 kilobase pairs upstream to 8 kilobase pairs downstream of unique transcription start sites, with overlapping spans merged via BEDTools.
Key facts
- Architecture
- ModernBERT encoder
- Parameter count
- 377M
- Pre-training corpus size
- 353,574,093,776 base pairs (443 vertebrate genome assemblies)
- Primary task
- Fill-mask (masked language modelling)
- Positional encodings
- Rotary Position Embeddings (RoPE)
What you can use it for
- Masked nucleotide prediction
- Predicting masked base positions in vertebrate DNA inputs through the masked language modelling fill-mask pipeline.
- Genomic sequence classification
- Fine-tuning pre-trained sequence representations on classification tasks using downstream task heads.
- Comparative genomics benchmarking
- Serving as a baseline model against other architectures on genomic evaluations such as the Nucleotide Transformer benchmark.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
ModernGENA large
Strengths and limits
Good at
- Integrates ModernBERT structural updates, including RoPE, GeGLU activations, and hybrid local/global attention.
- Offers high inference throughput through end-to-end unpadding, FlashAttention implementations, and torch.compile support.
Watch out for
- Pre-training excludes sequences with ambiguous nucleotide characters beyond A, C, G, and T.
- Sampling is concentrated in regions around transcription start sites rather than uniformly across the entire genome.
- Adapting the model to sequence classification requires downstream fine-tuning scripts.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use ModernGENA large commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run ModernGENA large without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
What tokenizer does ModernGENA large use?
It uses the 32k BPE tokenizer from AIRI-Institute/gena-lm-bert-base-t2t, which covers the characters A, T, G, C, and N along with standard BERT special tokens.
Can the model be fine-tuned for downstream biological tasks?
Yes, the pre-trained encoder can be loaded and fine-tuned for sequence classification tasks.
How to cite it
Back to BERT in 2026: ModernGENA as a Strong, Efficient Baseline for DNA Foundation Models
Written from ModernGENA large's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.