Literature · HuggingFaceBio
Carbon-500M
A 500M-parameter autoregressive DNA language model designed primarily as a draft model for speculative decoding.
What Carbon-500M is
Carbon-500M is a compact, decoder-only genomic language model developed through a collaboration between Hugging Face, Zhongguancun Academy, and TIGEM/University of Naples "Federico II". Containing 500 million parameters, it is a lightweight variant of the Carbon model family built to process both genetic sequences and natural language.
The model is primarily designed to act as a draft model in speculative decoding pipelines paired with larger models such as Carbon-3B and Carbon-8B. Because it shares the exact tokenizer and sequence format of the larger family members, it accelerates wall-clock generation times with zero loss in output quality compared to running the target model alone.
In addition to accelerating inference, the model is compact enough for downstream fine-tuning on consumer hardware. It supports base-pair-level decoding and sequence scoring through Factorised Nucleotide Supervision.
How it works
Carbon-500M uses a Llama-style decoder-only transformer architecture with Rotary Position Embeddings (RoPE), trained in bfloat16 precision using the AdamW optimizer. It features a native context window of 8,192 tokens (approximately 49 kilobase pairs) without a long-context extension stage. Sequence inputs are processed via a hybrid tokenizer combining 6-mer tokens for DNA (where each token represents roughly six base pairs) and Qwen3 byte-pair encoding (BPE) for English text.
The model was pre-trained on 600 billion 6-mer tokens (equivalent to roughly 3.6 trillion base pairs). Its training mixture remained constant across the entire run, consisting of 50% eukaryotic genes, 25% mature mRNA, 10% splice-enriched mRNA, and 15% GTDB bacterial genomes. Training began with standard cross-entropy loss for the first 300 billion tokens before switching to a hybrid Factorised Nucleotide Supervision (FNS) loss for the remaining 300 billion tokens, which enables nucleotide-level marginal probability estimation despite 6-mer tokenisation.
Key facts
- Architecture
- Decoder-only transformer (Llama-style, LlamaForCausalLM) with RoPE
- Parameter count
- 500 million
- Context length
- 8,192 tokens (approximately 49 kbp)
- Training data volume
- 600 billion tokens (approximately 3.6 trillion base pairs)
- Precision
- bfloat16
What you can use it for
- Speculative Decoding
- Serves as an assistant draft model alongside Carbon-3B or Carbon-8B to reduce latency and generation costs during DNA sequence generation without altering output quality.
- Base-Pair-Level Sequence Scoring
- Calculates per-nucleotide marginal probabilities and mean log-likelihoods to compare reference and perturbed genomic sequences using the Factorised Nucleotide Supervision revision.
- Lightweight Genomic Fine-Tuning
- Provides a compact foundation model suitable for parameter-efficient or full-model fine-tuning on specialized genomic tasks using consumer-grade hardware.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Carbon-500M
Strengths and limits
Good at
- Serves natively as an assistant model for Carbon-3B and Carbon-8B speculative decoding because they share an identical tokenizer and prompt format.
- Allows fine-grained, nucleotide-level sequence generation and scoring via its Factorised Nucleotide Supervision mechanism while retaining 6-mer tokenisation efficiency.
- Requires only 16 GB of GPU memory, making it accessible for execution and fine-tuning on consumer-grade hardware.
Watch out for
- Downstream task performance is constrained by its 500-million-parameter scale, making it uncompetitive with Carbon-3B or Carbon-8B as a standalone model.
- Context window is strictly bounded to its native 8,192 tokens (approximately 49 kbp) due to the absence of a long-context training stage.
- Factorised nucleotide-level generation and scoring requires running a custom code branch rather than standard model weights.
Inputs and outputs
You provide
Prompt
The model continues from here.
You get back
The model's completion.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use Carbon-500M commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run Carbon-500M without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
What is the primary role of Carbon-500M?
It is intended primarily as an assistant draft model for speculative decoding paired with larger target models like Carbon-3B or Carbon-8B, cutting inference time while guaranteeing identical generation results.
How does the model represent DNA and text sequences?
It uses a hybrid tokenizer that represents DNA sequences as 6-mers (each token representing roughly six base pairs) and text using Qwen3 byte-pair encoding.
Can Carbon-500M generate DNA at single base-pair resolution?
Yes, using the Factorised Nucleotide Supervision (FNS) revision, each 6-mer token is decomposed into six per-position nucleotide distributions to allow exact base-pair counts and per-nucleotide constraints.
What data was used to train the model?
The model was trained on 600 billion tokens comprising 50% eukaryotic genes, 25% mature mRNA, 10% splice-enriched mRNA, and 15% GTDB bacterial genomes.
How to cite it
Allal, Loubna Ben and Li, Qiuyi and Fiusco, Maurizio and Tunstall, Lewis and Rasul, Kashif and Beeching, Ed and Aubakirova, Dana and Patiño, Carlos and Frere, Thibaud and Lozhkov, Anton and others. Carbon: Decoding the Language of Life. bioRxiv, 2026--05, 2026. Cold Spring Harbor Laboratory.
Written from Carbon-500M's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.