Materials Chemistry Benchmark · jablonkagroup
MaCBench
A multimodal benchmark assessing vision large language models across chemistry and materials science.
What MaCBench is
MaCBench (Materials Chemistry Benchmark) is an evaluation benchmark developed by the Jablonka Group to probe the multimodal capabilities of vision-language models across chemistry and materials science. It evaluates how well models handle complex scientific imagery, laboratory practices, and domain-specific data interpretation.
General-purpose vision-language models often struggle with scientific diagrams, crystallography structures, and experimental spectra that require domain-specific reasoning rather than everyday object recognition. MaCBench addresses this by assembling tasks that require the joint interpretation of visual inputs and scientific questions, spanning from multiple-choice tests to open-ended quantitative reasoning.
The dataset contains over 1,100 questions manually curated by domain experts. It is strictly reserved for model evaluation and testing, including canary strings to prevent contamination in future training sets.
How it was made
MaCBench is structured into 35 configurations divided into three overarching domains: data extraction, in silico and lab experiments, and data interpretation. Every question contains both a text query and an associated image that must be interpreted together to deduce the answer.
The benchmark accommodates both multiple-choice and open-ended questions. Standardised evaluation metrics are assigned per task: multiple-choice items use multiple-choice grade scoring, while open-ended numerical or text tasks are scored using exact string matching, mean absolute error (MAE), and mean squared error (MSE), occasionally evaluated within a specified relative numerical tolerance.
Key facts
- Dataset size
- Over 1,100 questions across 35 configurations
- Input modalities
- Text and images
- Output modalities
- Text (multiple-choice selections or numerical/string responses)
- Evaluation metrics
- multiple_choice_grade, exact_string_match, mae, mse
- Format
- Parquet
- Primary fields
- uuid, image, canary, name, description, keywords, preferred_score, metrics, examples, relative_tolerance
What you can use it for
- Organic Structure and Reaction Extraction
- Test a model's ability to interpret hand-drawn molecules, extract reaction parameters (solvents, yield, temperature) from reaction schemas without SMILES strings, and assign stereochemical configurations or isomer types.
- Crystallographic and Laboratory Reasoning
- Assess whether vision models can identify unit-cell properties, point groups, crystal systems, and densities from crystal structure renderings, or recognise laboratory equipment and protocol violations from photographs.
- Spectroscopic and Isotherm Interpretation
- Evaluate model accuracy when extracting adsorption properties (such as Henry's constants and working capacity in metal-organic frameworks) from isotherms, reading bandgap types from electronic structures, or analysing XRD, MS, and NMR spectra.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from jablonkagroup/MaCBench at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- All questions are manually curated by chemistry and materials science specialists.
- Every evaluation item couples a text question with an image that must be visually analysed to determine the correct response.
- Covers a broad spectrum of scientific disciplines including organic chemistry, crystallography, surface characterisation, and spectroscopy.
- Includes canary strings within the dataset records to guard against model pre-training contamination.
Watch out for
- Restricted strictly to evaluation and benchmarking; it cannot be used for model training or fine-tuning.
- Monolingual dataset limited entirely to English text queries.
Questions
Can I look inside MaCBench without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is MaCBench under?
Its authors declare mit. Read the terms on the original page before reusing the data.
Can MaCBench be used to train or fine-tune multimodal models?
No. The dataset is explicitly designated for testing and evaluation only. Using the dataset for model training invalidates benchmark results and compromises the evaluation.
What formats and types of images are present?
The dataset contains diverse scientific imagery, including organic structure drawings, reaction schemes, patent figures and plots, laboratory photographs, 3D crystal structure renderings, AFM topography, XRD patterns, and spectral plots.
How are open-ended questions evaluated compared to multiple-choice items?
Multiple-choice items are evaluated using a categorical grade metric based on targeted choice scores. Open-ended items are evaluated using exact string matching, mean absolute error (MAE), or mean squared error (MSE), optionally incorporating a relative tolerance parameter for numerical targets.
What framework is used to execute the benchmark?
MaCBench is natively designed to integrate with the ChemBench evaluation engine, though it can also be evaluated with custom pipelines using Hugging Face datasets, pandas, or polars.
How to cite it
Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, Kevin Maik Jablonka. Probing the limitations of multimodal language models for chemistry and materials research. arXiv preprint arXiv: 2411.16955 (2024).
Written from MaCBench's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.