Literature · FreedomIntelligence
HuatuoGPT-Vision 7B
A multimodal medical model that answers clinical questions across radiology, pathology, and dermatology imagery.
What HuatuoGPT-Vision 7B is
HuatuoGPT-Vision 7B is a multimodal language model designed for clinical and medical applications. Developed by FreedomIntelligence as part of the HuatuoGPT family, it addresses the challenge of injecting large-scale medical visual knowledge into generative vision-language models.
The model assists researchers and clinicians by processing medical images alongside natural language queries, generating textual responses across diverse diagnostic fields including radiology, pathology, and dermatology. It provides conversational reasoning across both English and Chinese clinical contexts.
How it works
The model is constructed using the LLaVA-v1.5 architecture built upon a Qwen2-7B base. It is trained on PubMedVision, a dedicated medical visual dataset designed to scale visual knowledge injection across multimodal language backbones.
Key facts
- Architecture
- LLaVA-v1.5 with Qwen2-7B base (llava_qwen2)
- Parameters
- 7 billion
- Training dataset
- PubMedVision
- Supported languages
- English, Chinese
- Release date
- June 2024
- GPU memory requirement
- 40 GB
What you can use it for
- Radiological Question Answering
- Interpreting radiology scans alongside textual questions to assist in answering diagnostic and descriptive clinical inquiries.
- Pathology Image Analysis
- Evaluating pathology imagery to respond to queries regarding tissue samples and visual findings.
- Dermatology Visual Interpretation
- Answering clinical questions regarding dermatological photographs and presentations.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
HuatuoGPT-Vision 7B
Strengths and limits
Good at
- Trained specifically on the domain-focused PubMedVision dataset to integrate medical visual knowledge.
- Handles multimodal queries across multiple medical disciplines, including radiology, pathology, and dermatology.
- Supports both English and Chinese conversational inputs and completions.
- Builds upon the open LLaVA-v1.5 framework and Qwen2-7B foundation.
Watch out for
- Requires substantial computational resources, specifying 40 GB of GPU memory for serving.
- Generated outputs must be verified against primary medical documentation before being relied upon.
Inputs and outputs
You provide
Prompt
The model continues from here.
You get back
The model's completion.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use HuatuoGPT-Vision 7B commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run HuatuoGPT-Vision 7B without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on 1× A100 40GB, typically in 12s, and a run costs about 210 credits. Visitors without an account get 1,000 credits a month.
What image modalities does HuatuoGPT-Vision 7B support?
The model is built to answer clinical questions over medical imaging including radiology, pathology, and dermatology.
What is the underlying model architecture?
It is based on the Qwen2-7B backbone implemented within the LLaVA-v1.5 vision-language framework.
What dataset was used during training?
It was trained using the PubMedVision dataset curated by FreedomIntelligence.
Which human languages does the model support?
The model supports both English and Chinese text queries.
How to cite it
Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, Benyou Wang. (2024). HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale. arXiv:2406.19280.
Written from HuatuoGPT-Vision 7B's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.