Literature · HuggingFaceBio
Carbon-8B
An 8B-parameter autoregressive DNA model for genomic prediction, generation, and long-context sequence retrieval.
What Carbon-8B is
Carbon-8B is an 8-billion-parameter autoregressive DNA language model created through a collaboration between Hugging Face, Zhongguancun Academy, and TIGEM/University of Naples Federico II. It is designed for computational genomics, focusing primarily on eukaryotic DNA and RNA sequences to assist with sequence generation, variant effect assessment, and long-range motif analysis.
The model provides an expanded parameter capacity within the Carbon model family, tailored for researchers requiring higher accuracy on complex genomic prediction tasks. It operates on a large native context window and uses specialized supervision to evaluate sequences down to individual base pairs.
How it works
Carbon-8B employs a decoder-only causal transformer architecture (LlamaForCausalLM). Genomic inputs are tokenised using a hybrid approach combining non-overlapping 6-mers for DNA, where each token corresponds to six base pairs, alongside a Qwen3 byte-pair encoding vocabulary for text. Pre-training was conducted on 1 trillion 6-mer tokens (equivalent to roughly 6 trillion base pairs) drawn from a mixture containing 70% eukaryotic genomic DNA with metadata dropout, 16% messenger RNA, 4% spliced messenger RNA, and 10% prokaryotic sequence data.
Training used bfloat16 precision and the AdamW optimiser with a cosine learning-rate schedule. While training starts with standard cross-entropy loss, it shifts after 100 billion tokens to Factorized Nucleotide Supervision (FNS). This objective assembles each 6-mer from six per-nucleotide distributions, granting per-base resolution for marginal scoring and decoding.
Key facts
- Architecture
- Autoregressive decoder-only transformer (LlamaForCausalLM)
- Parameter count
- 8 billion
- Tokeniser
- Hybrid non-overlapping DNA 6-mer (6 bp/token) and Qwen3 BPE text tokeniser
- Native context length
- 32,768 tokens (approximately 196 kbp)
- Training volume
- 1 trillion 6-mer tokens (approximately 6 trillion base pairs)
- GPU memory requirement
- 40 GB
What you can use it for
- Variant effect prediction
- Evaluating the functional impact of single-nucleotide and non-coding alterations by calculating and comparing base-pair log-likelihood scores across clinical benchmarks like ClinVar and BRCA2.
- Base-pair-level DNA generation
- Synthesising genomic sequences with exact length constraints and nucleotide-level sampling control via the Factorized Nucleotide Supervision branch.
- Long-range genomic retrieval
- Identifying and retrieving specific functional motifs across large chromosomal contexts spanning up to 786 kilobase pairs.
- Sequence perturbation analysis
- Assessing the fitness and likelihood impacts of biological sequence modifications, such as nucleotide triplet expansions and synonymous codon replacements.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Carbon-8B
Strengths and limits
Good at
- Factorized Nucleotide Supervision provides nucleotide-level probability scoring and per-base decoding while preserving the speed of a 6-mer tokeniser.
- Achieves higher zero-shot accuracy than smaller family variants across eukaryotic sequence recovery, Mendelian trait prediction, and triplet-expansion perturbation tests.
- Trained on an extensive eukaryotic dataset incorporating diverse genomic elements, mRNA, and spliced transcripts.
Watch out for
- Requires substantial GPU memory, with inference demanding roughly 40 GB of VRAM.
- Standard decoding without the custom supervision branch operates over 4,096-way 6-mer token distributions rather than single base pairs.
- Long-context sequence retrieval performance declines past the 32,768-token native boundary unless extrapolation techniques such as YaRN are applied.
Inputs and outputs
You provide
Prompt
The model continues from here.
You get back
The model's completion.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use Carbon-8B commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run Carbon-8B without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on 1× A100 40GB, typically in 12s, and a run costs about 210 credits. Visitors without an account get 1,000 credits a month.
How does Carbon-8B score individual base pairs when using a 6-mer tokeniser?
Through Factorized Nucleotide Supervision, the model predicts six per-position nucleotide marginal distributions for each 6-mer token. A dedicated scoring function exposes the exact likelihood assigned to the observed base at every position.
What is the maximum sequence length supported by the model?
Carbon-8B natively handles up to 32,768 tokens, which equates to roughly 196 kilobase pairs.
What data mixture was used to train the model?
The pre-training mixture comprised 1 trillion 6-mer tokens split into 70% eukaryotic genomic sequence with metadata dropout, 16% mRNA, 4% spliced mRNA, and 10% prokaryotic DNA.
How does Carbon-8B differ from Carbon-3B?
Carbon-8B has higher parameter capacity, uses a cosine learning-rate schedule rather than a WSD schedule, and delivers improved performance on variant prediction, perturbation, and sequence recovery benchmarks at the expense of higher memory and compute requirements.
How to cite it
Allal, Loubna Ben, Qiuyi Li, Maurizio Fiusco, Lewis Tunstall, Kashif Rasul, Ed Beeching, Dana Aubakirova, Carlos Patiño, Thibaud Frere, Anton Lozhkov, and others. "Carbon: Decoding the Language of Life." bioRxiv (2026): 2026-05.
Written from Carbon-8B's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.