Genomics & Single-Cell · InstaDeep / Wellcome Sanger
Nucleotide Transformer v2
A DNA foundation model that generates nucleotide sequence representations for regulatory element and variant analysis.
What Nucleotide Transformer v2 is
Nucleotide Transformer v2 is a 500-million-parameter genomic foundation model developed by InstaDeep, NVIDIA, TUM, and the Wellcome Sanger Institute. Pretrained across 850 diverse genomes, it learns biological sequence representations from unannotated DNA to support molecular phenotype prediction, regulatory sequence identification, and variant scoring.
Rather than relying on a single human reference genome, the model incorporates genomic data from a broad spectrum of model and non-model organisms. It converts nucleotide sequences into high-dimensional vector embeddings that transfer effectively to downstream tasks such as promoter classification, enhancer detection, and splice site prediction with lightweight fine-tuning.
How it works
The model is a transformer that uses rotary positional embeddings and Gated Linear Units. DNA sequences are tokenised into 6-mers (six-nucleotide segments) where possible, falling back to individual nucleotides when necessary, across a vocabulary of 4,105 tokens. The maximum supported sequence length is 2,048 tokens.
Training relies on a BERT-style masked language modelling objective in which 15% of tokens are masked: 80% are replaced with a mask token, 10% are swapped for a random token, and 10% are left unchanged. The model was trained across eight 80 GB A100 GPUs on 900 billion tokens with an effective batch size of 1 million tokens using the Adam optimizer. The pretraining corpus comprises 850 NCBI genomes covering 174 billion nucleotides (approximately 29 billion tokens), explicitly excluding plants and viruses.
Key facts
- Parameter count
- 500 million
- Architecture
- Transformer with rotary positional embeddings and Gated Linear Units
- Context length
- 2,048 tokens
- Vocabulary size
- 4,105 tokens
- Training data
- 850 genomes from NCBI (174 billion nucleotides / ~29 billion tokens)
- Required GPU memory
- 16 GB
What you can use it for
- Promoter classification
- Classifying promoter sequences and activity from raw nucleotide sequences using sequence embeddings with minimal downstream fine-tuning.
- Enhancer identification
- Locating and predicting genomic enhancer regions across nucleotide sequences.
- Splice site prediction
- Identifying donor and acceptor splice sites across genomic sequences using pretrained representations.
- Variant effect scoring
- Assessing the molecular impact of genetic variants by evaluating changes in sequence embeddings and masked nucleotide likelihoods.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Nucleotide Transformer v2
Strengths and limits
Good at
- Pretrained across 850 diverse model and non-model genomes covering 174 billion nucleotides.
- Employs rotary positional embeddings and Gated Linear Units for robust sequence representation.
- Produces representations that adapt effectively to varied regulatory element classification tasks with light fine-tuning.
Watch out for
- Excludes plant and viral genomes from the pretraining data, limiting applicability to those organisms.
- Restricted to a maximum input sequence window of 2,048 tokens.
- Inference time required to compute embeddings increases significantly as input sequence length increases.
Inputs and outputs
You provide
DNA sequence
You get back
Sequence representation for downstream heads.
Licence
Academic and non-commercial use only · CC-BY-NC-SA-4.0
The CC-BY-NC-SA-4.0 licence does not permit commercial use of this model.
Questions
Can I use Nucleotide Transformer v2 commercially?
Academic and non-commercial use only. The CC-BY-NC-SA-4.0 licence does not permit commercial use of this model.
How do I run Nucleotide Transformer v2 without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on 1× A100 16GB, typically in 1.5s, and a run costs about 14 credits. Visitors without an account get 1,000 credits a month.
What types of genomes were used to pretrain the model?
The model was trained on 850 genomes from NCBI representing diverse model and non-model organisms, excluding plants and viruses due to differences in their regulatory elements.
How does the tokenizer process DNA sequences?
The tokenizer splits sequences into 6-mers wherever possible, tokenising remaining nucleotides individually, across a total vocabulary of 4,105 tokens.
What outputs does the model produce?
The model produces token-level hidden states, mean-pooled sequence embeddings for downstream prediction heads, and logits for masked nucleotide prediction.
What hardware is needed for inference?
The model specifies a requirement of 16 GB of GPU memory.
How to cite it
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Henryk Grywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard, et al. The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv (2023), 2023-01.
Written from Nucleotide Transformer v2's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.