ProteinGym

What ProteinGym is

ProteinGym is a benchmark suite designed for evaluating computational predictors of protein fitness and mutation effects. Developed by the OATML and Marks lab and presented at NeurIPS 2023, it provides a unified evaluation platform for comparing protein language models, inverse folding models, and supervised fitness predictors across both zero-shot and supervised settings.

The dataset spans four distinct tracks covering experimental and clinical contexts. These comprise deep mutational scanning (DMS) substitution assays measuring missense mutations, DMS indel assays measuring insertions and deletions, and two clinical tracks incorporating expert annotations from ClinVar and benign controls from gnomAD to assess clinical utility alongside experimental correlation.

Packaged on the Hub by LiteFold in Parquet format, this version provides unified access to nearly 3 million records. It offers standardized cross-validation regimes, precomputed multiple sequence alignments, wild-type sequences, and structural metadata to facilitate rigorous model benchmarking.

How it was made

The benchmark compiles experimental measurements and clinical annotations across diverse protein types, including enzymes, viral proteins, human disease genes, antibodies, and de novo designs. For each assay, records contain the wild-type sequence, per-variant experimental or clinical fitness values, precomputed multiple sequence alignments from ColabFold and EVE, and available structural annotations. Cross-validation splits are arranged under multiple schemes including random, modulo, and contiguous partition strategies.

This Hub packaging structures the data into a tabular Parquet schema with 118 top-level columns, consisting of common identifiers, provenance tags, normalized fields (such as sequences, mutations, targets, and scores), and the 100 most frequent raw fields formatted as snake_case strings. A complete raw JSON payload is preserved in each row, alongside auxiliary metadata tables mapping original columns and source table row counts. Data is partitioned into train (2,637,988 rows) and test (293,551 rows) sets using deterministic SHA-256 hash bucket allocation.

Key facts

What you can use it for

Zero-shot protein language model benchmarking

Assess unsupervised sequence models such as ESM, Tranception, or ProGen on their ability to rank mutation fitness without task-specific training.

Clinical variant pathogenicity prediction

Evaluate whether computational predictors can distinguish pathogenic mutations from benign controls using curated ClinVar and gnomAD benchmarks.

Inverse folding and structure-based fitness evaluation

Test structure-conditioned models using target structural metadata and experimental deep mutational scanning assays.

Indel mutation effect assessment

Measure predictor performance specifically on insertion and deletion variants across around 70 dedicated assays.

Explore the data

The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.

Strengths and limits

Questions

How to cite it

Notin, Pascal and Kollasch, Aaron W. and Ritter, Daniel and van Niekerk, Lood and Paul, Steffanie and Spinner, Hansen and Rollins, Nathan and Shaw, Ada and Orenbuch, Rose and Weitzman, Ruben and Frazer, Jonathan and Dias, Mafalda and Franceschi, Dinko and Gal, Yarin and Marks, Debora S. (2023). ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. Advances in Neural Information Processing Systems 36: Datasets and Benchmarks Track. doi:10.1101/2023.12.07.570727