Multi-Modal Foundation · google

MedSigLIP

A dual-encoder vision-language model mapping medical images and text into a shared embedding space for healthcare AI.

What MedSigLIP is

MedSigLIP is a multimodal dual-encoder foundation model developed by Google to encode medical imagery and text into a shared vector space. Built for healthcare machine learning tasks that do not require open-ended text generation, it supports data-efficient classification, zero-shot classification, and cross-modal semantic search across multiple clinical domains.

The model addresses the challenge of building compute-efficient visual understanding tools across varied medical specialties. It serves as an independent embedding tool and acts as the vision encoder powering image interpretation in the generative MedGemma model. Pre-training on clinical data combined with natural images enables the model to interpret specialised medical visuals while retaining broad visual comprehension.

How it works

MedSigLIP employs a two-tower architecture derived from SigLIP (Sigmoid Loss for Language Image Pre-training). Visual inputs are resized to 448 by 448 pixels and normalised between -1 and 1, while textual inputs are processed up to a maximum context length of 64 tokens. The system outputs separate image embeddings, text embeddings, and similarity logits.

Training was conducted in JAX using sigmoid loss on diverse de-identified medical image-text pairs alongside natural image-text data. Datasets spanned chest radiographs, whole-slide histopathology, retinal fundus imaging, teledermatology, and CT/MRI slices. Public sources incorporated MIMIC-CXR, Slake-VQA, PAD-UFES-20, SCIN, TCGA, CAMELYON, PMC-OA, MedQA, and Mendeley Digital Knee X-Ray, complemented by licensed clinical biobank and outpatient diagnostic imaging collections.

Key facts

Model architecture: Two-tower SigLIP (Vision Transformer and Text Transformer)
Image resolution: 448 x 448 pixels
Context length: 64 tokens
Training framework: JAX
Model version: 1.0.0
Release date: 9 July 2025

What you can use it for

Zero-shot medical image classification: Classify clinical images such as chest X-rays or skin lesions against prompt label descriptions without task-specific fine-tuning.
Cross-modal semantic retrieval: Query large diagnostic imaging repositories using natural language descriptions by measuring cosine alignment in the joint embedding space.
Data-efficient transfer learning: Train linear probes or shallow downstream classification heads directly on frozen visual embeddings using small labelled medical datasets.
Multimodal feature extraction: Extract high-dimensional numerical representations from pathology whole-slide patches, fundus photographs, and CT or MRI slices for downstream analytical pipelines.

Run it here

We list this model but cannot run it yet: excluded (gated): HAI-DEF terms; medical-imaging model..

Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.

Strengths and limits

Good at

Demonstrates competitive performance across radiology, pathology, dermatology, and ophthalmology benchmarks, often exceeding domain-specific embedding baselines.
Combines medical pre-training with natural image-text data to preserve general visual understanding alongside clinical feature extraction.

Watch out for

Does not generate natural language text, making it unsuitable for clinical reporting without a separate generative decoder like MedGemma.
Constrained to a maximum text sequence length of 64 tokens, limiting the detail of input prompts and descriptions.
Outputs must not be used directly for clinical diagnosis or treatment planning without task-specific adaptation and independent regulatory validation.
Performance assessments may carry contamination risks if public evaluation benchmarks overlapped with pre-training datasets.

Inputs and outputs

You provide

Image
PNG, JPEG or TIFF. Sent to the model as-is.
Candidate labels
Comma-separated. The model scores the image against each.

You get back

Labels with confidence scores, highest first.

Licence

Check the licence before commercial use · other

This model is gated by its authors — its licence has to be accepted on the source repository before it will run.

Questions

Can I use MedSigLIP commercially?

Check the licence before commercial use. This model is gated by its authors — its licence has to be accepted on the source repository before it will run.

Can MedSigLIP generate written radiology or pathology reports?

No. MedSigLIP is an encoder-only model that produces numerical embeddings and similarity scores between text and images. It does not perform autoregressive text generation.

What input formats and dimensions are required?

Images must be resized to 448 by 448 pixels and normalised to the range (-1, 1). Accompanying text inputs can be formatted as strings up to 64 tokens in length.

Which clinical imaging modalities were included in training?

The training corpus included chest radiographs, dermatological photography, ophthalmological fundus images, histopathology whole-slide images, and cross-sectional slices from CT and MRI scans.

How is MedSigLIP related to MedGemma?

MedSigLIP serves as the underlying vision encoder that provides image interpretation capabilities within the MedGemma generative vision-language architecture.

How to cite it

Sellergren, Andrew, et al. "MedGemma Technical Report." arXiv preprint arXiv:2507.05201 (2025).

Written from MedSigLIP's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.