Genomics & Single-Cell · Zhou et al., Northwestern
DNABERT-2
A byte-pair encoded transformer model generating nucleotide embeddings for genomic sequence classification.
What DNABERT-2 is
DNABERT-2 is a transformer-based genome foundation model developed by Zhou and colleagues at Northwestern University. It addresses the computational overhead and tokenisation limitations of earlier genomic transformers by replacing fixed k-mer tokenisation with byte-pair encoding (BPE). The model is designed to process DNA sequences and produce dense vector representations suitable for downstream genomic prediction tasks.
Trained on multi-species genome data, the model serves as an efficient baseline for a broad range of sequence-level analyses. Across the Genome Understanding Evaluation (GUE) benchmark suite, this architecture delivers improved speed and performance compared to predecessor models, offering an accessible tool for computational biologists studying genome organisation and regulatory syntax.
How it works
DNABERT-2 builds upon the MosaicBERT transformer architecture. Instead of fragmenting nucleotide sequences into overlapping k-mers—which substantially increases sequence length and computational cost—the model applies byte-pair encoding to nucleotide sequences. This compresses the input sequence into variable-length sub-words.
The transformer processes these encoded token sequences and generates per-token hidden states of dimension 768. Researchers can aggregate these sequence representations using pooling strategies, such as mean pooling or max pooling, to obtain fixed-dimensional sequence-level embeddings for classification.
Key facts
- Architecture
- MosaicBERT-based transformer with Byte-Pair Encoding
- Parameter count
- 117 million
- Training data
- Multi-species genome
- Input modality
- DNA nucleotide sequence
- Output modality
- 768-dimensional embedding vector
- GPU memory required
- 8 GB
What you can use it for
- Regulatory Sequence Classification
- Generate sequence-level embeddings to classify functional regulatory regions and promoter elements across diverse species.
- Motif Discovery
- Analyse sequence representations to identify recurring functional motifs within unannotated genomic regions.
- Splice Site Prediction
- Classify donor and acceptor splice junctions directly from genomic nucleotide sequences.
- General Sequence Benchmarking
- Serve as a lightweight baseline model for classification benchmarks across the Genome Understanding Evaluation suite.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
DNABERT-2
Strengths and limits
Good at
- Uses byte-pair encoding instead of k-mer tokenisation, resulting in faster throughput and reduced token sequence lengths.
- Demonstrates improved performance across tasks in the Genome Understanding Evaluation (GUE) benchmark suite relative to k-mer models.
- Operates as a computationally economical genomic baseline with modest hardware requirements of 8 GB GPU memory.
- Outputs standard 768-dimensional embeddings that support both mean and max pooling strategies.
Watch out for
- Requires downstream pooling or classification heads to convert token-level hidden states into specific biological predictions.
- Demands dedicated GPU memory (minimum 8 GB) to compute representations efficiently.
Inputs and outputs
You provide
DNA sequence
You get back
Sequence representation.
Licence
Commercial use permitted · Apache-2.0
The Apache-2.0 licence allows use in commercial research and products.
Questions
Can I use DNABERT-2 commercially?
Commercial use permitted. The Apache-2.0 licence allows use in commercial research and products.
How do I run DNABERT-2 without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on 1× A100 16GB, typically in 800ms, and a run costs about 7 credits. Visitors without an account get 1,000 credits a month.
What is the primary difference between DNABERT and DNABERT-2?
DNABERT-2 replaces the overlapping k-mer tokenisation of the original DNABERT with byte-pair encoding (BPE) and adopts the MosaicBERT architecture, which improves both processing speed and benchmark accuracy.
What input format does the model require?
The model takes raw DNA nucleotide sequences containing standard nucleotide characters (A, C, G, T) and converts them into token identifiers using its dedicated BPE tokeniser.
What output does DNABERT-2 produce?
It returns a sequence of hidden state representations with a hidden dimension of 768, which can be pooled across the sequence dimension using mean or max operations to yield a single 768-dimensional vector.
On what data was the model trained?
The model was trained on genome sequences sampled from multiple species.
How to cite it
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, Han Liu. DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome. arXiv:2306.15006, 2023.
Written from DNABERT-2's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.