Multi-Modal Foundation · AIRI-Institute
ModernGENA base
A 377M-parameter ModernBERT DNA foundation model for genomic sequence modelling and downstream analysis tasks.
What ModernGENA base is
ModernGENA base is a genomic foundation model developed by AIRI-Institute that adapts the ModernBERT encoder architecture for DNA sequence modelling. Containing 377 million parameters, the model serves as an efficient baseline designed for downstream genomic applications, including sequence classification, variant analysis, and gene expression studies.
The architecture addresses computational demands in genomics by pairing modernized transformer components with support for long sequence contexts. It provides accelerated inference throughput and delivers competitive results on standard benchmarks, such as the Nucleotide Transformer benchmark, while remaining practical for comparative architectural research.
How it works
The model builds upon a ModernBERT encoder structure incorporating pre-norm transformer blocks, Rotary Position Embeddings (RoPE), hybrid local and global attention, and GeGLU feed-forward layers. It employs end-to-end unpadding and is compatible with FlashAttention and compiler optimizations for accelerated processing.
The corpus incorporates both forward and reverse complement strands, excluding ambiguous characters other than standard bases and N. To prioritise informative regulatory regions over simple repeats, training data was sampled in windows spanning 16 kilobase pairs upstream to 8 kilobase pairs downstream (-16 kbp to +8 kbp) of unique transcription start sites, with overlapping spans merged using BEDTools.
Key facts
- Architecture
- ModernBERT encoder (RoPE, GeGLU, hybrid local/global attention)
- Parameter count
- 377M
- Pre-training data
- 443 vertebrate genome assemblies (353,574,093,776 bp)
What you can use it for
- Genomic sequence classification
- Fine-tuning the pre-trained encoder to categorise regulatory regions or other functional DNA segments.
- Variant effect prediction
- Evaluating the functional impact of genomic alterations in downstream variant analysis tasks.
- Expression modelling
- Supplying contextual representations of promoter-proximal regions for downstream gene expression tasks.
- Masked nucleotide prediction
- Predicting masked base pairs within DNA sequences using the fill-mask objective.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
ModernGENA base
Strengths and limits
Good at
- High inference throughput enabled by FlashAttention and end-to-end unpadding.
- Demonstrated competitive performance on the Nucleotide Transformer benchmark.
- Modernized encoder design combining Rotary Position Embeddings, GeGLU units, and hybrid attention.
- Pre-training focused on transcription start site intervals across diverse vertebrate assemblies.
Watch out for
- Pre-training sequences omit ambiguous IUPAC genomic characters other than A, C, G, T, and N.
- Dataset sampling is restricted to promoter-proximal windows rather than unbiased whole-genome tiling.
- Inference acceleration requires hardware and software configurations compatible with FlashAttention.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use ModernGENA base commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run ModernGENA base without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
What architecture does ModernGENA base use?
It is based on the ModernBERT encoder architecture, using Rotary Position Embeddings, pre-norm transformer blocks, GeGLU feed-forward networks, and hybrid local/global attention.
Which benchmarks has the model been evaluated against?
ModernGENA base was evaluated on downstream tasks within the Nucleotide Transformer benchmark.
How to cite it
Back to BERT in 2026: ModernGENA as a Strong, Efficient Baseline for DNA Foundation Models
Written from ModernGENA base's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.