Research Benchmark · futurehouse
BixBench
A benchmark of 205 reproducible scientific questions paired with data capsules to evaluate AI systems.
What BixBench is
BixBench is an evaluation benchmark developed by FutureHouse designed to test the capabilities of artificial intelligence systems on scientific reasoning. It spans domains including biology, chemistry, and mathematics.
The benchmark comprises 205 reproducible research questions, each associated with an individual data capsule to support open-answer and task-specific AI evaluation.
How it was made
The benchmark is structured as a flattened text dataset stored in JSONL format, where each row contains one question. Each entry corresponds to an accompanying data capsule packaged as a compressed ZIP file titled with a specific identifier (CapsuleFolder-{uuid}.zip).
Key facts
- Total questions
- 205
- Data format
- JSONL and ZIP
- Subject domains
- Biology, chemistry, mathematics, scientific reasoning
- Current versions
- v1.0, v1.5
What you can use it for
- AI scientific reasoning evaluation
- Assessing the performance of machine learning models on complex, reproducible research tasks across biology, chemistry, and mathematics.
- Open-answer response testing
- Evaluating model capabilities when answering open-ended scientific questions provided alongside dedicated data capsules.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from futurehouse/BixBench at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- Pairs reproducible research questions directly with dedicated data capsules in individual zip archives.
- Features a flattened single-question-per-row format refined for answerability in open-answer contexts.
Watch out for
- Contains a limited sample size of 205 research questions.
- The original release (v1.0) contained questions with insufficient detail to be readily answerable in open-ended settings.
Questions
Can I look inside BixBench without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is BixBench under?
Its authors declare apache-2.0. Read the terms on the original page before reusing the data.
What formats are the dataset files provided in?
The core questions are provided in a BixBench.jsonl file, while the associated data capsules are packaged individually as CapsuleFolder-{uuid}.zip archives.
What is the difference between version 1.0 and version 1.5?
Version 1.5 flattens the dataset to one question per row and revises a substantial portion of questions to ensure they provide sufficient detail for open-answer evaluation, whereas version 1.0 retains the original question set.
What scientific domains are represented?
The benchmark covers biology, chemistry, mathematics, and general scientific reasoning.
Written from BixBench's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.