Scientific Benchmark · google
spiqa
A multimodal benchmark for evaluating question answering across scientific paper figures, tables, and text.
What spiqa is
SPIQA (Scientific Paper Image Question Answering) is a benchmark designed to assess how well multimodal large language models comprehend scientific documents. Published scientific literature relies heavily on complex visual presentations—including schematic diagrams, experimental plots, result visualisations, and data tables—alongside explanatory prose. Answering questions about these papers requires models to reason across both visual and textual modalities simultaneously.
It provides resources to test multiple interaction paradigms, including answering directly from figures and tables, reading the complete paper text alongside figures, or carrying out retrieval before answering.
How it was made
The dataset is compiled from papers gathered from arXiv and divided into training, validation, and three distinct test splits (test-A, test-B, and test-C). Questions and answers for the training, validation, and test-A splits were generated using multimodal models; test-A underwent additional manual verification and curation. In contrast, test-B and test-C derive from the existing QASA and QASPER benchmarks, respectively, providing human-authored questions and answers filtered to require multimodal interpretation.
Each entry is indexed by its arXiv identifier and stores metadata including captions, extracted figure classifications (such as schematics, plots and charts, visualisations, or tables), questions, reference rationales, and pointers to source visual and textual evidence. Papers are accompanied by full-resolution PNG images, downscaled 224-pixel versions for test splits, extracted body paragraphs, and raw LaTeX source files.
Key facts
- Release date
- June 2024
- Visual categories
- Schematics, plots and charts, visualisations, other figures, tables
- Input formats
- JSON metadata, PNG images, extracted text paragraphs, raw TeX files
- Supported tasks
- Direct QA with figures/tables, Direct QA with full paper, Chain-of-Thought (CoT) retrieval QA
- Language
- English
What you can use it for
- Multimodal Model Benchmarking
- Evaluate the visual reasoning and document understanding capabilities of vision-language models on complex scientific diagrams and statistical charts.
- Scientific Document Retrieval
- Test retrieval-augmented generation pipelines on their ability to retrieve the specific figure, table, or paragraph necessary to answer technical questions.
- Chain-of-Thought Reasoning Evaluation
- Measure multi-step reasoning performance where a system must first identify relevant figures and tables from a paper before composing an answer.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from google/spiqa at the moment you asked. Nothing is cached or stored — every row above came from that request.
How the columns are distributed
Over 666 rows of default/test
answer
string_text- min
- 1
- median
- 110
- mean
- 150.77
- max
- 966
paper_id
string_label- 1805.04687v23%
- 1805.06431v43%
- 1811.09393v42%
question
string_text- min
- 36
- median
- 89
- mean
- 100.08
- max
- 341
rationale
string_text- min
- 60
- median
- 288
- mean
- 306.78
- max
- 918
reference_figure
string_text- min
- 25
- median
- 26
- mean
- 25.73
- max
- 28
Strengths and limits
Good at
- Categorises visual assets into distinct types including schematics, plots, photos, and tables.
- Features human-authored test sets (test-B and test-C) derived from QASA and QASPER alongside curated automated sets.
- Supplies both full-resolution images and 224-pixel versions for testing under different computational budgets.
Watch out for
- The vast majority of questions in the training, validation, and test-A sets are machine-generated rather than human-written.
- Source literature is restricted to computer science conference papers deposited on arXiv between 2018 and 2023.
- Only English-language scientific papers are included.
Questions
Can I look inside spiqa without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is spiqa under?
Its authors declare cc-by-4.0. Read the terms on the original page before reusing the data.
What is the difference between test-A, test-B, and test-C?
Test-A contains machine-generated questions that were manually filtered and curated. Test-B and test-C consist of human-authored questions adapted and filtered from the QASA and QASPER datasets, respectively.
What types of visual elements are included in the dataset?
The dataset categorises visual elements into schematics, plots and charts, result visualisations, tables, and miscellaneous figures, all extracted from arXiv paper sources.
What task formats can be evaluated using this benchmark?
The benchmark supports direct question answering given specific figures and tables, direct question answering using the entire paper context, and chain-of-thought question answering where systems must first retrieve helpful visual elements before answering.
How to cite it
Pramanick, S., Chellappa, R. and Venugopalan, S., 2024. SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers. NeurIPS. arXiv:2407.09413.
Written from spiqa's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.