Multi-Modal Foundation · wanglab
MedSAM
A promptable medical image segmentation model fine-tuned on CT, MRI, X-ray, ultrasound, and histology data.
What MedSAM is
MedSAM is a foundation model developed by wanglab for medical image segmentation. Built upon the Segment Anything Model (SAM) architecture with a Vision Transformer Base (ViT-Base) backbone, it adapts universal promptable segmentation to biomedical imaging.
The model addresses the challenge of segmenting anatomical structures and pathological targets across diverse clinical imaging modalities. It has been fine-tuned on a collection of computed tomography (CT), magnetic resonance imaging (MRI), radiography (X-ray), ultrasound, and histology imagery, allowing researchers and clinicians to extract masks using spatial prompts.
How it works
MedSAM consists of a vision encoder, a prompt encoder, a mask decoder, and a prediction neck. The VisionEncoder is a Vision Transformer that computes image embeddings using self-attention over image patches along with relative positional embeddings. The PromptEncoder converts user-supplied prompts, such as spatial points or bounding boxes, into feature embeddings.
The MaskDecoder is a two-way transformer that conducts bidirectional cross-attention between image embeddings and prompt embeddings. A neck module then computes the final segmentation masks based on the contextualised representations produced by the decoder.
Key facts
- Architecture
- Vision Transformer Base (ViT-Base) with prompt encoder and mask decoder
- Input prompt types
- Points and bounding boxes
- Supported modalities
- CT, MRI, X-ray, ultrasound, histology
- Primary task
- Medical image mask generation
What you can use it for
- Radiological Image Segmentation
- Extracting anatomical and pathological segmentations from CT, MRI, and X-ray images using point or box prompts.
- Ultrasound Target Delineation
- Generating object masks for structures within ultrasound scans based on spatial guidance.
- Histology Tissue Segmentation
- Segmenting histological structures and cellular regions in microscopic pathology images.
Run it here
We list this model but cannot run it yet: loads, then fails at inference — see FAILS_AT_INFERENCE.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Fine-tuned across diverse medical modalities including CT, MRI, X-ray, ultrasound, and histology.
- Accepts flexible spatial prompting via coordinates and bounding boxes to define segmentation targets.
- Employs bidirectional cross-attention between image and prompt embeddings to generate contextualised masks.
Watch out for
- Relies on prompt inputs, such as points or bounding boxes, to isolate specific structures of interest.
- Training coverage is specific to the documented modalities of CT, MRI, X-ray, ultrasound, and histology.
Inputs and outputs
You provide
Image
PNG, JPEG or TIFF. Sent to the model as-is.
You get back
The returned image.
Licence
Check the licence before commercial use · apache-2.0
The apache-2.0 licence carries terms worth reading before you depend on it.
Questions
Can I use MedSAM commercially?
Check the licence before commercial use. The apache-2.0 licence carries terms worth reading before you depend on it.
What prompts can be passed to MedSAM?
MedSAM accepts points and bounding boxes as input prompts to specify target structures for mask generation.
Which imaging modalities does the model support?
The model is fine-tuned on CT, MRI, X-ray, ultrasound, and histology images.
What is the model architecture?
MedSAM is built on the Segment Anything Model ViT-Base architecture, containing a ViT vision encoder with relative positional embeddings, a prompt encoder, a two-way transformer mask decoder, and a prediction neck.
How to cite it
Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Dollar, Piotr and Girshick, Ross. (2023). Segment Anything. arXiv:2304.02643.
Written from MedSAM's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.