Literature · HuggingFaceBio
Carbon-3B
A 3-billion-parameter autoregressive DNA foundation model for sequence generation and variant-effect prediction.
What Carbon-3B is
Carbon-3B is an autoregressive decoder-only genomic foundation model developed by Hugging Face Biology Research in collaboration with Zhongguancun Academy and TIGEM/University of Naples "Federico II". Built to model eukaryotic DNA and RNA sequences, it targets biological sequence generation, variant interpretation, and genomic perturbation analysis.
The model employs a 6-mer tokenisation strategy where each token represents six base pairs, allowing it to cover extensive genomic sequences efficiently. It provides a native context length of 32,768 tokens (approximately 197 kilobase pairs) and can be extended to 65,536 tokens (approximately 393 kilobase pairs) at inference time.
Carbon-3B achieves competitive accuracy with larger single-nucleotide models on zero-shot variant-effect prediction, sequence recovery, and sequence-level perturbation tasks, while maintaining high generation throughput on modern accelerator hardware.
How it works
Carbon-3B uses a Llama-style Transformer architecture featuring 30 layers, a hidden dimension of 3072, an intermediate feed-forward size of 8448, SwiGLU activation functions, RMSNorm, and grouped-query attention with 32 attention heads and 4 key-value groups. It relies on rotary position embeddings (RoPE) and a hybrid vocabulary that combines non-overlapping 6-mer tokens for DNA sequences with a Byte-Pair Encoding (BPE) text vocabulary.
The model was pre-trained on 1 trillion 6-mer tokens (equivalent to roughly 6 trillion base pairs) across two primary phases. Phase 1 used standard cross-entropy loss over 100 billion tokens, after which Phase 2 transitioned to Factorised Nucleotide Supervision (FNS), an objective that bridges coarse 6-mer tokenisation with single-nucleotide resolution. The pre-training corpus predominantly features eukaryotic genomic DNA, complemented by mature mRNA, splice-enriched mRNA, and bacterial genomes from the Genome Taxonomy Database (GTDB), followed by a 50-billion-token long-context extension phase.
Key facts
- Architecture
- Llama-style decoder-only Transformer with SwiGLU, RMSNorm, and GQA
- Parameters
- 3 billion
- Layers
- 30
- Hidden size
- 3072
- Attention heads
- 32 attention heads (4 key-value groups)
- Native context length
- 32,768 6-mer tokens (≈ 197 kbp)
- Extended context length
- 65,536 6-mer tokens (≈ 393 kbp) via YaRN
- Training data volume
- 1 trillion 6-mer tokens (≈ 6 trillion base pairs)
What you can use it for
- Variant-effect prediction
- Evaluating the functional impact or pathogenicity of single-nucleotide and structural variants across coding and non-coding regions using likelihood-based log-probability scoring.
- Generative sequence design
- Generating synthetic DNA sequences either unconditionally or conditionally guided by prepended species-type and gene-type metadata tokens.
- Sequence perturbation discrimination
- Distinguishing natural genomic references from perturbed sequences, including nucleotide triplet expansions in coding regions and synonymous codon replacements.
- Long-range sequence retrieval
- Locating and retrieving specific distal regulatory motifs or target sequence patterns across contexts spanning up to 393 kilobase pairs.
- Base-pair-level decoding and scoring
- Using the Factorised Nucleotide Supervision branch to evaluate per-nucleotide marginal probabilities and generate sequences with exact nucleotide-level control.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Carbon-3B
Strengths and limits
Good at
- High inference efficiency, reaching generation speeds exceeding 100,000 base pairs per second on a single H100 GPU when using vLLM.
- Native long-context handling of approximately 197 kilobase pairs, extendable to roughly 393 kilobase pairs via YaRN.
- Supports lossless speculative decoding when paired with Carbon-500M as a draft model.
- Matches the performance of larger single-nucleotide baselines like Evo2-7B on benchmarks such as ClinVar non-coding and BRCA2.
Watch out for
- DNA inputs must be enclosed in specific sequence tags; passing untagged sequences causes fallback to text BPE tokenisation and sharp performance drops.
- Only uppercase A, C, G, and T characters are accepted in DNA mode, mapping lowercase bases and IUPAC ambiguity codes to unknown tokens.
- Input sequences that are not multiples of six base pairs are right-padded with adenine bases by the 6-mer tokenizer unless pre-truncated.
- Long-context retrieval accuracy degrades significantly when using YaRN scaling beyond 65,536 tokens.
- The training distribution is heavily skewed toward eukaryotes, requiring further training for specialised prokaryotic applications.
Inputs and outputs
You provide
Prompt
The model continues from here.
You get back
The model's completion.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use Carbon-3B commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run Carbon-3B without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on 1× A100 16GB, typically in 12s, and a run costs about 70 credits. Visitors without an account get 1,000 credits a month.
How are DNA sequences processed by the tokenizer?
Carbon-3B uses non-overlapping 6-mer tokens, grouping every six base pairs into a single token. Sequences must be wrapped with DNA tags to engage 6-mer mode; otherwise, the tokenizer defaults to standard text BPE.
How can the context window be extended beyond 32k tokens?
The context window can be expanded to 65,536 tokens (around 393 kilobase pairs) at inference time by applying YaRN rope scaling with an expansion factor of 4.0 in the model configuration.
Can Carbon-3B perform single-base resolution generation?
Yes, using the Factorised Nucleotide Supervision (FNS) branch, the model unpacks 6-mer predictions into individual per-position nucleotide distributions for single-base-pair generation and scoring.
Does the model support conditional DNA generation?
Yes, the model was pre-trained with metadata tokens, permitting generation to be conditioned on specific species-type and gene-type identifiers.
How to cite it
Allal, L. B., Li, Q., Fiusco, M., Tunstall, L., Rasul, K., Beeching, E., Aubakirova, D., Patiño, C., Frere, T., Lozhkov, A., et al. (2026). Carbon: Decoding the Language of Life. bioRxiv, 2026-05. Cold Spring Harbor Laboratory.
Written from Carbon-3B's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.