Mathematical Reasoning · SAIRfoundation
equational-theories-benchmark
A mathematical reasoning benchmark evaluating whether language models can determine equational implications over magmas.
What equational-theories-benchmark is
The equational-theories-benchmark is a mathematical evaluation suite built by the SAIRfoundation for Stage 1 of the Mathematics Distillation Challenge: Equational Theories. It measures the formal reasoning capabilities of artificial intelligence models on problems drawn from abstract algebra, specifically equational implication over magmas (sets equipped with a single binary operation without required axioms such as associativity).
Given two algebraic equations, the benchmark task requires a model to decide whether the first equation implies the second. The benchmark standardises evaluations across multiple problem subsets and reasoning configurations to allow systematic comparison of model correctness, consistency, and resource efficiency.
How it was made
The benchmark compiles evaluations across six distinct configurations formed by pairing three problem subsets (hard, normal, and hard2) with two reasoning settings: a low-reasoning setting (low or no reasoning effort with low temperature) and a default-reasoning setting (default reasoning effort with default temperature). Each setting evaluates 200 selected problems against 25 shared language models, with 3 independent evaluation runs executed per model-problem pair using a common evaluation prompt template without cheatsheets.
The suite is structured into multiple data configurations. The source-of-truth table ('runs') documents each individual run, recording the raw response, token counts, latency, monetary cost, correctness, and judgment reason. The 'cells' table aggregates metrics across the three repeats for each model-problem pair (including majority and consensus correctness), while the 'leaderboard' table provides summary metrics per model, strictly classifying unparsed outputs as false negatives or false positives based on the target label.
Key facts
- Task
- Equational implication over magmas (Equation 1 implies Equation 2)
- Number of benchmarks
- 6 configurations
- Problems per benchmark
- 200
- Models evaluated
- 25 shared models
- Repeats per cell
- 3 runs per model/problem pair
- Data format
- Tabular and text (JSONL, CSV)
- Size category
- 100K to 1M items
What you can use it for
- Model reasoning evaluation
- Assess and compare how well general language and reasoning models deduce formal implications between algebraic equations without domain-specific cheatsheets.
- Consistency and stability testing
- Measure the variance and repeat consistency of model outputs across multiple temperature and reasoning configurations over identical algebraic problems.
- Inference cost-efficiency analysis
- Evaluate trade-offs between model accuracy, token counts, elapsed execution time, and financial cost across reasoning regimes.
Explore the data
No licence declared
The authors have not published a licence, which means no permission to redistribute the data has been granted to anyone. The rows are not served here; read them on the original dataset page.How the columns are distributed
Over 6 rows of benchmarks/train
benchmark_id
string_label- normal_200_common_25_default_reason17%
- hard_200_common_25_default_reason17%
- hard_200_common_25_low_reason17%
cheatsheet_mode
string_label- none100%
model_count
int- min
- 25
- median
- 25
- mean
- 25
- max
- 25
problem_count
int- min
- 200
- median
- 200
- mean
- 200
- max
- 200
problem_source_dataset
string_label- SAIRfoundation/equational-theories-selected-problems100%
problem_subset
string_label- hard233%
- normal33%
- hard33%
reasoning_mode
string_label- low_or_none50%
- default50%
repeat_count
int- min
- 3
- median
- 3
- mean
- 3
- max
- 3
temperature_mode
string_label- low50%
- default50%
template_id
string_label- evaluation100%
template_path
string_label- prompts/evaluation.jinja2100%
Strengths and limits
Good at
- Tests models against a standardised set of 25 shared models across all six benchmark configurations.
- Provides granular execution records including raw outputs, parsed judgment rationales, token counts, execution times, and compute costs.
- Applies strict evaluation scoring where unparsed outputs are penalised directly in confusion matrix counts and F1 scores rather than ignored.
Watch out for
- Task scope is restricted exclusively to equational implications over magmas rather than broader algebraic structures.
- Benchmark leaderboard metrics rely solely on parsed judgment summaries and do not use raw response text as a fallback when parsing fails.
- Evaluations are limited to the three problem sets ('hard', 'normal', and 'hard2') consisting of 200 problems each.
Questions
Can I look inside equational-theories-benchmark without downloading it?
Not here: it declares no licence, so no permission to redistribute its rows has been granted. Its description and statistics are still shown.
What licence is equational-theories-benchmark under?
Its authors have not declared a licence, which means no permission to reuse or redistribute it has been granted.
What is the primary scientific task in this benchmark?
The task is equational implication over magmas. Models are presented with two algebraic equations (Equation 1 and Equation 2) and must determine whether Equation 1 mathematically implies Equation 2.
What problem subsets are evaluated?
The suite covers three problem subsets: 'normal', 'hard', and 'hard2', with 200 problems included in each subset.
How are unparsed model responses treated during scoring?
Unparsed responses are scored strictly: an unparsed run on a TRUE-labeled problem counts as a false negative, while an unparsed run on a FALSE-labeled problem counts as a false positive.
How are the six benchmark settings configured?
The six settings combine the three problem subsets ('normal', 'hard', and 'hard2') with two operational modes: 'low_reason' (low or no reasoning effort with low temperature) and 'default_reason' (default reasoning mode with default temperature).
Written from equational-theories-benchmark's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.