Medical Benchmark · openai
healthbench
A medical benchmark evaluating AI assistants on multi-turn conversations using physician-written rubrics.
What healthbench is
HealthBench is an open evaluation benchmark created by OpenAI to assess artificial intelligence assistants in healthcare and clinical settings. It focuses on realistic, multi-turn health conversations, providing standardised conversational data to measure performance in applied medical dialogues.
The dataset addresses the challenge of systematically evaluating conversational AI in medicine. It provides structured conversation turns scored against rubrics designed by physicians across dimensions including factual accuracy, completeness of information, and communication quality.
How it was made
The benchmark comprises multi-turn dialogues representing healthcare interactions. Responses are assessed against physician-written grading rubrics that evaluate specific performance axes, including clinical accuracy, completeness, and communication standards.
Key facts
- Category
- Medical Benchmark
- Fields
- Medicine, benchmark, scientific reasoning
- Evaluation dimensions
- Accuracy, completeness, communication
What you can use it for
- Medical Assistant Evaluation
- Assessing how accurately conversational artificial intelligence models handle health-related queries across multi-turn interactions.
- Clinical Communication Scoring
- Grading dialogue models against physician-written criteria to evaluate clinical communication quality and completeness.
- Scientific Reasoning Assessment
- Testing the multi-turn reasoning capabilities of healthcare-focused language models in simulated patient or clinical dialogues.
Explore the data
The schema, real rows — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from openai/healthbench at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- Features realistic multi-turn conversations reflecting complex, sequential health interactions.
- Evaluations are based on rubrics formulated by qualified physicians.
- Measures performance across multiple critical clinical axes, including accuracy, completeness, and communication.
Watch out for
- Covers evaluation data and rubrics, requiring external evaluation frameworks such as simple-evals for complete pipeline execution.
Questions
Can I look inside healthbench without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser. Nothing is downloaded to your machine.
What licence is healthbench under?
Its authors declare mit. Read the terms on the original page before reusing the data.
What axes are evaluated in HealthBench?
Models are evaluated across clinical accuracy, completeness of information, and communication quality.
Who created the grading criteria for HealthBench?
The evaluation rubrics were written by physicians.
Does HealthBench consist of single-turn prompts or dialogues?
It consists of realistic multi-turn health conversations rather than isolated single-turn queries.
Written from healthbench's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.