Biology Benchmark · Anthropic
BioMysteryBench-preview
A preview benchmark evaluating AI scientific reasoning through anonymised bioinformatics analysis tasks.
What BioMysteryBench-preview is
BioMysteryBench-preview is an evaluation benchmark developed by Anthropic designed to test artificial intelligence on complex, real-world biological and bioinformatics reasoning. It serves as a five-problem preview slice of the larger, gated ninety-problem BioMysteryBench suite.
Evaluating computational models on genuine scientific problem-solving is difficult because models can often memorise published literature or look up public accession codes. To counter this, each task supplies anonymised biological data files alongside a scientific question that requires hands-on analysis to resolve, preventing simple dataset retrieval.
The problems address topics across structural biology, microbiology, genomics, and transcriptomics. Benchmark tasks are designed so that original experimental studies cannot simply be identified from public metadata, requiring models or human researchers to execute genuine bioinformatics investigations.
How it was made
The dataset consists of a problems metadata table and an archive of raw data files. The table contains problem identifiers, the scientific questions, evaluation rubrics, lists of allowed domains, and indicators of whether each task is known to be solvable by human experts. The archive supplies the underlying data files, which are extracted into an analysis environment to run computational workflows.
Evaluation protocol strictly prohibits querying accession numbers from repositories such as the Gene Expression Omnibus (GEO), Sequence Read Archive (SRA), European Nucleotide Archive (ENA), or BioProject to uncover source publications. However, standard reference lookups and analytical tools—including reference genome downloads, gene identifier resolution, sequence annotation, and BLAST searches—are permitted to perform the biological investigations.
Key facts
- Number of problems
- 5
- Full benchmark size
- 90 problems
- Problem fields
- id, question, answer_rubric, allowed_domains, human_solvable
- Data formats
- CSV, Parquet, ZIP
- Release version
- v11 (2026-07)
What you can use it for
- Evaluating AI Scientific Reasoning
- Test the autonomous biological reasoning and computational problem-solving abilities of AI models on genuine bioinformatics tasks.
- Organism Identification from Structural Data
- Analyse unlabelled macromolecular crystal structure files to deduce and report the source organism using standard binomial nomenclature.
- Microbial Metagenomics and Sequencing Analysis
- Inspect raw sequencing files to detect bacterial species or quantify distinct tissue groupings in non-human microbial community datasets.
- Functional Genomics and Gene Regulation Inference
- Process ChIP-seq peak datasets to determine transcription factor binding identities or analyse gene sets to identify associated microRNAs.
Explore the data
No queryable preview
This dataset is not available in a queryable form — usually an unusual file format or a repository too large to convert. The original source may still be readable.Strengths and limits
Good at
- Problems provide anonymised data files that require authentic bioinformatics execution rather than simple literature memorisation or lookup.
- Includes explicit evaluation rubrics and permitted domain guidelines for standardised assessment.
- Labels tasks by human solvability, distinguishing problems achievable by human experts from those that are not.
Watch out for
- Contains only five preview problems, whereas the full benchmark comprises ninety problems.
- Strict benchmark rules disallow looking up accession numbers across GEO, SRA, ENA, and BioProject databases.
Questions
Can I look inside BioMysteryBench-preview without downloading it?
Not here: it is not available in a queryable form. Its description and statistics are still shown.
What licence is BioMysteryBench-preview under?
Its authors declare cc-by-4.0. Read the terms on the original page before reusing the data.
What is the difference between this preview and the full benchmark?
This preview contains five sample problems, while the full BioMysteryBench suite comprises ninety problems held in a separate, gated evaluation repository.
Are external databases allowed during problem solving?
Standard bioinformatics tools and databases—such as BLAST, reference genomes, gene ID lookups, and sequence annotation—are allowed. However, searching accession IDs on platforms like GEO, SRA, ENA, or BioProject to identify source papers is strictly prohibited.
What types of scientific data are included?
The problems provide anonymised data spanning protein crystal structures, sequencing datasets, transcription factor binding peaks, and gene lists associated with microRNAs and microbial communities.
Written from BioMysteryBench-preview's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.