Multi-Modal Foundation · microsoft
CoLiPRI
A 3D vision–language encoder that jointly represents chest CT scans and radiology reports for medical imaging research.
What CoLiPRI is
COLIPRI (Comprehensive Language–Image Pre-training) is a 3D vision–language transformer model developed by Microsoft Research. It is designed to encode volumetric chest computed tomography (CT) scans alongside unstructured radiological reports into a shared representation space. The released model weights correspond to the COLIPRI-CRM variant.
The model addresses the challenge of interpreting volumetric medical scans without relying on manual annotations. By training on tens of thousands of paired scans and clinical reports with multiple objectives, it learns joint representations of 3D radiology data and clinical text to support diverse downstream analytical workflows.
How it works
COLIPRI is structured as a 3D vision–language transformer encoder. The image encoder processes volumetric 3D CT scans—preprocessed to standardised spatial dimensions such as 192×192×192 voxels—and maps them to 768-dimensional representations. Depending on the task, it outputs spatial patch embeddings (sized 768×24×24×24) or pooled 768-dimensional vector representations. Paired radiological reports are processed to map image and text features into a shared embedding space.
Pre-training was conducted on Azure Machine Learning across four NVIDIA A100 (80 GB) GPUs for 72 hours (288 GPU-hours total). The training pipeline utilised nnSSL for self-supervised training, TorchIO for 3D data augmentation and preprocessing, and nifti-zarr-py for data loading.
Key facts
- Model type
- 3D vision–language encoder
- Model variant
- COLIPRI-CRM
- Training data
- Tens of thousands of unannotated chest CT scans and radiological reports
- Data origins
- Turkey and the United States
- Pre-training compute
- 4 × NVIDIA A100 (80 GB) for 72 hours (288 GPU-hours)
- Embedding dimensions
- 768
What you can use it for
- Zero-shot CT classification
- Classify 3D chest CT volumes against natural language text prompts without task-specific training, such as identifying the presence or absence of lung nodules.
- Cross-modal retrieval
- Perform text-to-image and image-to-text querying across chest CT repositories and associated radiological reports.
- Dense feature extraction
- Extract 3D spatial patch embeddings or pooled volume vectors to supply downstream segmentation decoders or classification layers.
- Unimodal clustering and search
- Group or search CT volumes and clinical reports using image-to-image or text-to-text similarity comparisons.
Run it here
We list this model but cannot run it yet: no loader for library: colipri.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Learns joint representations directly from paired 3D CT scans and text without manual clinical labelling.
- Enables zero-shot classification via prompt engineering without requiring fine-tuning.
- Produces both spatially resolved 3D patch embeddings and global pooled embeddings.
Watch out for
- Restricted strictly to research use and not permitted for clinical practice.
- Trained exclusively on patient data from Turkey and the United States, introducing possible demographic and geographic biases.
- Underlying biases present within the training data may not be fully characterised.
Inputs and outputs
You provide
Image
PNG, JPEG or TIFF. Sent to the model as-is.
Candidate labels
Comma-separated. The model scores the image against each.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · mit
The mit licence carries terms worth reading before you depend on it.
Questions
Can I use CoLiPRI commercially?
Check the licence before commercial use. The mit licence carries terms worth reading before you depend on it.
Can COLIPRI be deployed in diagnostic clinical workflows?
No. The model is released exclusively for research purposes and is not meant to be used for clinical practice.
Is fine-tuning required to adapt COLIPRI to downstream tasks?
Fine-tuning is typically unnecessary to achieve good performance in downstream tasks such as zero-shot classification or feature extraction.
What downstream architectures can be coupled with COLIPRI?
The encoders can be connected to external linear classifiers, segmentation decoders for dense image predictions, or language decoders for radiological report generation.
How to cite it
Wald, T., Hamamci, I. E., Gao, Y., Bond-Taylor, S., Sharma, H., Ilse, M., … Pérez-García, F. (2026). Comprehensive language–image pre-training for 3D medical image understanding. In Computer Vision – ECCV 2026 (pp. 388–406). Springer Nature Switzerland. doi:10.1007/978-3-032-37624-4_24
Written from CoLiPRI's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.