CryoET Benchmark · biohub
popsicle
A cryo-electron tomography benchmark for macromolecular localization and 3D cellular structure segmentation.
What popsicle is
POPSICLE (Particle/Object Picking and Segmentation In CryoET Learning Evaluation) is a benchmark suite designed for cryo-electron tomography (cryoET). Created by researchers at the Chan Zuckerberg Biohub and the CryoET Data Portal community, it standardises machine learning evaluation across dense voxel-wise segmentation of cellular structures and sparse 3D point localization of macromolecular complexes.
Cryo-electron tomography produces three-dimensional reconstructions of vitrified biological specimens, but existing machine learning benchmarks in this domain have traditionally been fragmented, task-specific, and small. POPSICLE addresses this by supplying a unified manifest layer over data hosted on the CryoET Data Portal, establishing standardized splits, canonical class taxonomies, ground-truth filtering, and evaluation metrics across eukaryotic, prokaryotic, and lysate preparations.
The benchmark allows researchers to compare computational vision architectures across differing supervision scales, ranging from low-data regimes with seven tomograms to corpora containing more than a thousand tomograms.
How it was made
Rather than distributing multi-terabyte raw volumes directly, POPSICLE operates as an indexing and manifest layer. Each sub-benchmark provides Croissant 1.1 JSON-LD manifests alongside tabular CSV index files that describe tomogram locations, voxel sizes, splits, and annotations residing in the CryoET Data Portal. Tooling compatible with the copick interface reads these manifests and streams tomogram slices and 3D ground-truth coordinates on demand directly from portal cloud storage.
The suite is split into four distinct sub-benchmarks: Phantom evaluates multi-class localization across six purified macromolecular targets in cellular lysate; Bacterial benchmarks dense compartment segmentation across eight prokaryotic genera; Yeast evaluates organelle segmentation in Schizosaccharomyces pombe; and MotorBench assesses single-class point localization of bacterial flagellar motors.
Key facts
- Modality
- 3D cryo-electron tomography (cryoET)
- Tasks
- Sparse 3D point localization and dense voxel-wise segmentation
- Data manifest format
- Croissant 1.1 JSON-LD and CSV
- Access interface
- copick library streaming from CryoET Data Portal
- Phantom dataset size
- 7 train, 121 validation, 364 test tomograms
- Bacterial dataset size
- 68 train, 12 test tomograms
- Yeast dataset size
- 16 train, 4 test tomograms
- MotorBench dataset size
- 1,559 train, 843 test tomograms
- Curator
- Biohub Dynamic Structural Biology / CryoET Data Portal community
What you can use it for
- Macromolecular particle picking
- Benchmarking deep learning detection models, such as DeepFinder or DeepETPicker, to accurately predict 3D coordinate centers for protein complexes across diverse molecular weights.
- Dense cellular compartment segmentation
- Training volumetric segmentation architectures, including nnU-Net, SwinUNETR, and MedNeXt, to partition intracellular regions, membranes, and organelles in bacteria and yeast.
- Multi-task vision model evaluation
- Assessing how well foundation models and promptable 3D architectures generalize between sparse localization regimes and dense voxel-level segmentation tasks.
Explore the data
The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.
Explore
Read the real rows without downloading anything
Reading rows…
LiveRead from biohub/popsicle at the moment you asked. Nothing is cached or stored — every row above came from that request.
Strengths and limits
Good at
- Unifies both sparse point localization and dense voxel-wise segmentation under a single standardized evaluation surface.
- Eliminates large local storage requirements by streaming 3D volumes dynamically using Croissant metadata and the copick toolkit.
- Spans multiple biological contexts, including eukaryotic cells, eight bacterial genera, and controlled macromolecular lysates.
Watch out for
- It is not designed for structure determination and does not provide refined particle orientations or aligned 3D subtomogram averages.
- The Phantom training split is intentionally limited to seven tomograms, making it unsuitable for studying parameter scaling curves.
- Tomograms are subject to the missing wedge artifact, creating anisotropic resolution and elongation along the optical axis.
- Reference annotations for beta-amylase have lower confidence due to weak contrast, requiring its exclusion from aggregate F4 scoring.
Questions
Can I look inside popsicle without downloading it?
Yes. This page reads its schema, real rows and per-column statistics in the browser, with full-text search and a SQL filter. Nothing is downloaded to your machine.
What licence is popsicle under?
Its authors declare cc0-1.0. Read the terms on the original page before reusing the data.
How are the raw tomograms and annotations retrieved?
POPSICLE distributes metadata manifests rather than raw voxel arrays. Using the copick library, users load the Croissant JSON-LD file, which resolves data URLs against the public CryoET Data Portal bucket and streams arrays into memory as NumPy arrays.
What particle classes are included in the Phantom localization benchmark?
Phantom contains six classes spanning approximately 268 to 4300 kDa: apo-ferritin, beta-amylase, beta-galactosidase, cytosolic ribosomes, thyroglobulin, and virus-like particles.
Which evaluation metrics are standard for POPSICLE localization tasks?
Localization benchmarks employ recall-weighted F-beta metrics, specifically F4 for Phantom and F2 for MotorBench, prioritising high target retrieval over false positive filtering.
What structures are annotated in the bacterial and yeast segmentation sets?
The bacterial benchmark includes cytosol, flagellum, dense body inclusions, intermembrane space, and membrane. The yeast benchmark covers cytoplasm, nucleus, nuclear envelope, vesicle, membrane-enclosed lumen tubules, and mitochondrion.
How to cite it
POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET. Advances in Neural Information Processing Systems, 2026.
Written from popsicle's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.