Multi-Modal Foundation · InstaDeepAI
Nucleotide Transformer 2.5B
A 2.5B-parameter DNA language model trained across 850 species genomes for molecular phenotype prediction.
What Nucleotide Transformer 2.5B is
Nucleotide Transformer 2.5B is a genomic foundation language model developed jointly by InstaDeep, NVIDIA, and the Technical University of Munich (TUM). While conventional DNA language models frequently rely on single reference genomes, this model leverages whole-genome DNA sequences from 850 diverse species, encompassing both model and non-model organisms.
The model addresses the challenge of predicting molecular phenotypes directly from DNA sequences. By learning representations across a broad evolutionary spectrum, it serves as a foundation for downstream biological tasks such as promoter, enhancer, and splice site identification.
Formulated as a masked language model, it can compute embeddings for arbitrary DNA sequences or fill masked nucleotides, enabling computational biologists to extract sequence representations for genomic analysis and molecular phenotype prediction.
How it works
The model is a 2.5-billion-parameter transformer trained using a BERT-style masked language modelling objective. DNA sequences are tokenised into 6-mers (sequences of six consecutive nucleotides) when possible, falling back to individual nucleotides otherwise, using a vocabulary of 4,105 tokens. During training, 15% of tokens are selected for masking: 80% are replaced with a mask token, 10% are replaced with a random alternative token, and 10% remain unchanged.
Pre-training was conducted on 128 NVIDIA A100 80GB GPUs over 300 billion tokens with an effective batch size of 1 million tokens and a context length of 1,000 tokens. The underlying dataset comprises 174 billion nucleotides (approximately 29 billion tokens) collected from 850 genomes from the NCBI database, intentionally excluding plants and viruses.
Key facts
- Parameter count
- 2.5 billion
- Architecture
- Transformer (BERT-style masked language model)
- Context length
- 1,000 tokens
- Tokenizer vocabulary size
- 4,105 tokens (6-mer based)
- Training data
- 850 genomes from NCBI (174 billion nucleotides / ~29 billion tokens)
- Pre-training volume
- 300 billion tokens
- Required GPU memory
- 16 GB
What you can use it for
- Promoter and enhancer prediction
- Identify and classify genomic regulatory elements by fine-tuning on labelled promoter and enhancer datasets or extracting sequence embeddings.
- Splice site identification
- Predict donor and acceptor splice junctions in multi-species genomic sequences.
- DNA sequence representation extraction
- Generate per-token and mean-pooled hidden-state sequence embeddings for downstream machine learning classifiers and genomic analyses.
- Masked nucleotide prediction
- Predict missing or masked nucleotides within a given genomic sequence using the fill-mask capability.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Nucleotide Transformer 2.5B
Strengths and limits
Good at
- Pre-trained across 850 diverse species genomes rather than relying on a single reference genome.
- Employs a 6-mer tokenisation strategy that balances sequence context length with nucleotide-level resolution.
- Demonstrates strong performance on benchmark genomic tasks including promoter, enhancer, and splice site prediction.
- Supports both PyTorch and TensorFlow implementations via Hugging Face Transformers.
Watch out for
- Genomes from plants and viruses were excluded from training data, limiting applicability to those clades.
- Maximum sequence length is constrained to 1,000 tokens per input pass.
- Embedding computation time increases significantly when sequences are padded to full maximum length.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use Nucleotide Transformer 2.5B commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run Nucleotide Transformer 2.5B without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
Which organisms are included in the pre-training data?
The model was pre-trained on 850 genomes from NCBI covering a wide range of model and non-model species, but plant and viral genomes were specifically excluded.
How are DNA sequences tokenised by the model?
Sequences are converted using a 6-mer tokenizer with a vocabulary of 4,105 tokens, segmenting sequences into 6-nucleotide chunks where possible and falling back to single nucleotides when necessary.
What is the maximum context length for input sequences?
The maximum input length is 1,000 tokens.
Can the model output both sequence embeddings and masked token probabilities?
Yes, it can run as a fill-mask pipeline to predict masked tokens and can also output token-level or mean-pooled hidden state embeddings.
How to cite it
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G., et al. (2023). The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv, 2023-01. Cold Spring Harbor Laboratory.
Written from Nucleotide Transformer 2.5B's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.