Demonstration only. Corollary is a non-commercial research demo — no warranty, no support, not for commercial use or clinical decisions. Read the disclaimer

Corollary Labs
All use cases

Literature and study design · What you can do here

Reproduce a published figure or benchmark

Test a specific published benchmark or quantitative figure by setting explicit tolerances beforehand, running the canonical code on isolated compute, and reporting an evidence-based verdict.

Start this in the research agent

What this involves

Reproducing a published result requires isolating a single claim, figure or benchmark from a paper and testing whether identical or equivalent computation yields the expected value. The research agent locates the official implementation and reference materials, verifies code and data provenance, and establishes quantitative success criteria before executing any scripts.

Doing this work properly means defining numerical tolerances, metric units, comparison directions and required validation controls in advance, rather than adjusting them after observing the outcome. The agent writes a structured reproduction plan to a dedicated working directory, logs environmental details and random seeds, runs canonical procedures first, and evaluates whether discrepancies stem from experimental variability, implementation bugs or setup failures.

All operations execute on a dedicated environment where tools clone repositories, inspect issues and run scripts using Python or R without requiring local installation. The resulting report records exact commands, data checksums and outputs alongside a strict categorical verdict: supported, partially supported, contradicted, or not tested.

How it goes

  1. 1Define evaluation criteria and tolerancesIdentify one specific result to test, then record the metric, expected value, units, uncertainty, comparison direction and justified numerical tolerance before executing code.
  2. 2Inspect repository and code provenanceSearch and inspect official repositories using github_search, github_repo, github_list_files and github_read_file, pinning exact tags or commits and checking licence terms.
  3. 3Plan and prepare the execution environmentClone the code to /work/<name> using run_command, install dependencies with conda or uv, and write a reproduction plan into /work/results/reproduction-<unique-id>/ recording seeds and dataset checksums.
  4. 4Execute canonical benchmarks and capture outputsRun the canonical procedure via run_python, run_r or background start_job, logging exit statuses, raw commands and provenance sidecars with output hashes.
  5. 5Compare results and assign a verdictCompare observed values against the predefined criterion and assign a verdict of SUPPORTED, PARTIALLY SUPPORTED, CONTRADICTED, or NOT TESTED alongside remaining uncertainties.

Start it here

Ask in plain language. The agent works on a Linux machine of its own — it installs and runs what the job needs, keeps every file, and reads the results back with the numbers explained. Nothing to set up, and a long run continues after you close the page.

“Reproduce a published figure or benchmark. A fixed success criterion, the inputs and code recorded, failures preserved, and a verdict backed by evidence.”
Open the research agent

What to watch for

  • Never loosen numerical tolerances after observing the computed result.
  • A crashed script must be classified as NOT TESTED rather than CONTRADICTED.
  • A plausible value alone does not constitute evidence of support.
  • Do not claim exact reproducibility solely from an unpinned branch name or a general package list.
  • A saved output file alone does not demonstrate reproducibility without verified commands and environment hashes.

Questions

Do I need to install anything to reproduce a published figure or benchmark?

No. The research agent does this work on a Linux machine of its own — it installs what the job needs, keeps the files, and reads the results back to you. Long runs continue if you close the page.

What verdicts can the reproduction workflow report?

The workflow reports one of four verdicts: SUPPORTED when criteria and checks pass, PARTIALLY SUPPORTED when a defined subset or trend matches, CONTRADICTED when a valid test conflicts with the original claim, or NOT TESTED when execution or setup fails.

How are published figures reproduced?

Quantitative figures are reproduced directly from code and original data rather than through automatic image generation, allowing numerical comparisons against expected values.

How does the agent handle missing repository dependencies or setup errors?

If missing inputs or dependencies prevent execution, the agent marks the test as NOT TESTED and documents the exact blocker and raw logs without generating substitute evidence.

Why must tolerances be defined before running the code?

Setting tolerances, units and comparison directions in advance prevents post-hoc adjustments and ensures that evaluation criteria remain objective.

Written from the method the research agent follows for this work and the documentation of the tools listed above. Statements citing a figure that does not appear there are removed before publication. Corollary is a non-commercial research demonstration; results are not for clinical use.

Other things you can do here