Chemistry Benchmark · futurehouse
ether0-benchmark
A chemistry reasoning benchmark for language models featuring tasks where all answers are molecular structures.
What ether0-benchmark is
The ether0-benchmark is a chemical question-answering evaluation dataset created by FutureHouse to assess reasoning in frontier language models. It targets core chemical problem-solving challenges by compiling tasks derived from standard chemical data sources, such as reaction prediction from the United States Patent and Trademark Office (USPTO) and the Open Reaction Database (ORD), molecular captioning from PubChem, and Globally Harmonized System (GHS) hazard classifications.
A defining characteristic of the benchmark is that all answers resolve to a molecule. The suite is balanced across tasks to provide a consistent test bed for computational chemistry systems, ensuring that models can be evaluated on chemical reasoning without requiring open-ended text evaluation.
How it was made
The benchmark compiles various chemical reasoning problems into a structured question-answering format stored as Parquet files. Problem designs are calibrated against established reference standards; for example, reaction prediction accuracy on the benchmark aligns with performance on withheld splits of the USPTO-50k dataset.
Evaluation tasks span multiple chemistry disciplines and use Simplified Molecular Input Line Entry System (SMILES) strings and molecular representations. Tasks include completing SMILES fragments, designing molecules satisfying specific formulas and functional group constraints, predicting forward reaction products, proposing one-step retrosynthetic pathways, altering molecular solubility, translating IUPAC names to SMILES, and answering multiple-choice questions concerning safety, toxicity, scent, pKa, and ADME (absorption, distribution, metabolism, and excretion) properties such as blood-brain barrier permeability.
Key facts
- File format
- Parquet
- Data modality
- Text
- Primary representations
- SMILES and IUPAC names
- Answer type
- Molecules
- Associated paper
- arXiv:2506.17238
What you can use it for
- Reaction Prediction and Synthesis Evaluation
- Benchmarking language models on forward reaction outcome prediction and single-step retrosynthetic pathway proposals.
- Constrained Molecular Generation
- Assessing a model's capacity to design molecules matching specified chemical formulas, complete broken SMILES fragments, and satisfy functional group rules.
- Chemical Nomenclature Translation
- Testing the ability of models to convert standard IUPAC chemical names into valid SMILES structures.
- Property and Safety Assessment
- Evaluating model predictions on solubility edits, pKa, scent, GHS hazard categories, toxicity, and pharmacokinetic ADME properties including blood-brain barrier permeability.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from futurehouse/ether0-benchmark at the moment you asked. Nothing is cached or stored — every row above came from that request.
How the columns are distributed
Over 325 rows of default/test
id
string_text- min
- 36
- median
- 36
- mean
- 36
- max
- 36
ideal
string_text- min
- 8
- median
- 30.50
- mean
- 39.59
- max
- 210
7.7% empty
problem
string_text- min
- 62
- median
- 176
- mean
- 234.49
- max
- 1,089
problem_type
string_text- min
- 13
- median
- 24
- mean
- 25.15
- max
- 72
solution
string_text- min
- 41
- median
- 76
- mean
- 85.67
- max
- 241
unformatted
string_text- min
- 6
- median
- 32
- mean
- 46.09
- max
- 380
15.4% empty
Strengths and limits
Good at
- Every evaluation task requires a molecule as the answer, providing a uniform output structure across disparate chemical domains.
- Balanced design of approximately 25 questions per task keeps evaluation practical and low-latency for large language models.
- Task outcomes are calibrated against established reference baselines, such as withheld test splits of USPTO-50k.
Watch out for
- It restricts evaluation strictly to questions that can be answered with a molecular structure, omitting open-ended chemical explanations.
Questions
Can I look inside ether0-benchmark without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is ether0-benchmark under?
Its authors declare cc-by-4.0. Read the terms on the original page before reusing the data.
What format do the answers take in this benchmark?
All target answers across the benchmark are molecules, typically represented via SMILES strings.
What chemical properties and safety metrics are tested?
The questions cover ADME properties including blood-brain barrier permeability, GHS safety classifications, toxicity, scent, pKa, and solubility modification.
How to cite it
arXiv:2506.17238
Written from ether0-benchmark's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.