Genomic Benchmark · HuggingFaceBio
perturbation-bench
Evaluates DNA foundation models on pairwise sequence discrimination with codon and CAG repeat perturbations.
What perturbation-bench is
Perturbation Bench is a genomic evaluation suite developed by HuggingFaceBio to measure how accurately DNA foundation models differentiate natural genomic sequences from artificially altered ones. The benchmark challenges models to assign higher log-likelihood values to authentic reference sequences than to structurally modified pairs across coding regions.
Evaluating sequence-level understanding is essential for determining whether genomic language models capture true biological signals such as natural codon preferences or disease-relevant motifs, rather than merely scoring raw sequence composition. The benchmark tests these properties across tens of thousands of gene loci in both human and mouse genomes.
How it was made
In the synonymous tasks, coding sequences (CDS) have their codons substituted with the target organism's highest-frequency synonym according to CoCoPUTs RefSeq frequencies, changing an average of 55% of codons while preserving the underlying amino acid sequence. In the motif task, a 30-base-pair codon-aligned sequence downstream of the first exon codon is replaced with ten consecutive CAG triplets, mimicking pathological polyglutamine repeat expansions.
Every sequence pair is presented within an 8,192-base-pair window, maintaining identical flanking regions and reading-frame alignments between the positive (original) and negative (perturbed) sequences. Benchmark performance is quantified as pairwise discrimination accuracy, calculated as the proportion of pairs where the model evaluates the log-likelihood of the unperturbed sequence to be higher than that of the perturbed sequence.
Key facts
- Window size
- 8,192 bp
- Task configurations
- syn_human, syn_mouse, motif_human
- Genomic builds
- GRCh38/hg38 (human) and GRCm39/mm39 (mouse)
- Annotation sources
- GENCODE v45, GENCODE vM34, and CoCoPUTs
- Evaluation metric
- Pairwise discrimination accuracy (mean(LL(original) > LL(perturbed)))
- Data format
- Parquet
What you can use it for
- Codon usage sensitivity benchmarking
- Assessing whether genomic foundation models favour natural codon distributions over artificially optimised synonymous variants across human and mouse coding regions.
- Trinucleotide repeat expansion detection
- Testing model capacity to penalise pathological CAG expansions that model polyglutamine disorders such as Huntington's disease and spinocerebellar ataxias.
- Cross-species model comparison
- Comparing the sequence-level discrimination capabilities of foundation models across human (GENCODE v45) and mouse (GENCODE vM34) genomic contexts.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from HuggingFaceBio/perturbation-bench at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- Preserves reading frames and total sequence lengths across all positive and negative pairs within consistent 8,192 bp windows.
- Isolates codon usage preference from protein-level changes by keeping translated amino acid sequences completely identical.
- Covers a broad biological scope comprising thousands of distinct human and mouse genes across all exon ranks.
Watch out for
- Constrained to fixed 8,192 bp context windows, which may not capture long-range genomic interactions beyond this span.
- Restricted to synonymous substitutions and CAG repeat expansions, omitting other classes of genomic structural variants.
- Requires models that support explicit log-likelihood computation for full-length input sequences.
Questions
Can I look inside perturbation-bench without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is perturbation-bench under?
Its authors declare apache-2.0. Read the terms on the original page before reusing the data.
How is performance measured on Perturbation Bench?
Performance is reported as pairwise discrimination accuracy, defined as the proportion of instances where a model assigns a higher log-likelihood to the real, unperturbed sequence than to the perturbed sequence.
What perturbation is introduced in the motif_human task?
The task replaces a 30 bp codon-aligned region located 60 bp downstream of the CDS exon start with ten consecutive CAG triplets, modelling the trinucleotide repeats found in polyglutamine disorders.
How were the synonymous codon replacements selected?
Codons were replaced using the highest-frequency synonym for each amino acid derived from CoCoPUTs species-specific RefSeq CDS frequencies, changing between 10% and 95% of codons per sequence.
How to cite it
Allal, L. B., Li, Q., Fiusco, M., Tunstall, L., Rasul, K., Beeching, E., Aubakirova, D., Patiño, C., Frere, T., Lozhkov, A., et al. (2026). Carbon: Decoding the Language of Life. bioRxiv, pages 2026-05. Cold Spring Harbor Laboratory.
Written from perturbation-bench's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.