STEM Benchmark · facebook
principia-bench
A curated benchmark evaluating language models on deriving mathematical objects from STEM problems.
What principia-bench is
Principia Bench is an evaluation benchmark developed by Facebook to assess how effectively language models derive mathematical objects from STEM-related problem statements. Covering disciplines including mathematics, physics, and chemistry, the collection tests multi-step reasoning, domain-specific scientific knowledge, and quantitative analysis.
The dataset aggregates 2,241 instances sourced from four existing benchmarks: RealMath, SuperGPQA, Physics, and ARB. Each record comprises a self-contained single-question prompt paired with a ground-truth answer that is formally expressed as a mathematical object.
By providing standardised mathematical targets rather than multiple-choice selections, the benchmark allows researchers to evaluate whether models can genuinely calculate and construct scientific solutions or broadly reason through natural science problems.
How it was made
The dataset was constructed by drawing candidate problems from RealMath, SuperGPQA (with original answer options removed), Physics, and ARB. The compilers filtered these sources using three criteria: the target answer must be expressed as a mathematical object, the problem statement must consist of only a single question, and the statement must be completely self-contained.
Evaluation can be carried out using model-based judgement guided by a specific judge prompt to compare generated responses against the ground-truth answers.
Key facts
- Total instances
- 2,241
- Scientific fields
- Mathematics, physics, chemistry
- Source datasets
- RealMath, SuperGPQA, Physics, ARB
- Data format
- JSON
- Language
- English
- Input format
- Text problem statement (single question)
- Output format
- Mathematical object
What you can use it for
- Mathematical Object Derivation
- Test the ability of language models to formulate and derive precise mathematical objects from scientific prompts.
- STEM Multi-Step Reasoning Evaluation
- Benchmark scientific reasoning across quantitative problems drawn from chemistry, physics, and mathematics.
- Open-Ended Problem Solving
- Assess model answers on problems where multiple-choice answer selections have been removed, requiring direct generation of the answer.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from facebook/principia-bench at the moment you asked. Nothing is cached or stored — every row above came from that request.
How the columns are distributed
Over 2,241 rows of default/train
answer
string_text- min
- 2
- median
- 49
- mean
- 74.47
- max
- 829
problem_statement
string_text- min
- 19
- median
- 232
- mean
- 261.14
- max
- 1,164
source_data
string_label- SuperGPQA65%
- RealMath28%
- Physics5%
Strengths and limits
Good at
- All 2,241 problems require deriving an explicit mathematical object rather than guessing multiple-choice options.
- Every problem is filtered to be self-contained and limited to a single question, avoiding multi-part ambiguities.
- Sourced from four diverse STEM benchmarks spanning mathematics, physics, and chemistry.
Watch out for
- Restricted only to problems where solutions can be expressed as mathematical objects.
- Relies on model-based judgement prompts for automated scoring of outputs.
Questions
Can I look inside principia-bench without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is principia-bench under?
Its authors declare cc-by-nc-4.0. Read the terms on the original page before reusing the data.
How many problems are included in Principia Bench?
The benchmark includes 2,241 instances.
Which benchmark datasets are included in the compilation?
Problems are drawn from RealMath, SuperGPQA, Physics, and ARB.
How are multiple-choice questions handled from source benchmarks?
For datasets like SuperGPQA, multiple-choice options are stripped away so models must derive the mathematical answer directly.
How are responses evaluated?
Responses are assessed against ground-truth answers using a model-based judgement prompt.
Written from principia-bench's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.