Multi-Modal Foundation · doctolib-lab
DoctoBERT-fr-base
A 111M-parameter French RoBERTa encoder pretrained for biomedical and clinical text processing.
What DoctoBERT-fr-base is
DoctoBERT-fr-base is an encoder model designed specifically for French biomedical and clinical natural language processing. Developed by doctolib-lab, it uses a RoBERTa architecture with 111 million parameters and a 512-token context window. The model addresses the challenge of understanding real-world French medical documentation, which frequently contains high medical term density, non-standard syntax, and variable registers.
Trained from scratch on large-scale French medical datasets, the model is designed to produce representations for clinical tasks such as named entity recognition, medical classification, and semantic retrieval. In standard benchmark evaluations across French and English medical language models, it attains top aggregate performance across clinical and biomedical tasks.
How it works
The model follows a standard 12-layer RoBERTa configuration with 12 attention heads, a hidden dimensionality of 768, and an intermediate feed-forward size of 3072 using Gaussian Error Linear Unit (GELU) activations. Text is tokenised using a 32,768-token SentencePiece Byte-Pair Encoding (BPE) vocabulary trained on filtered medical documents. The tokeniser splits numerical digits into individual tokens and incorporates byte-level fallback for unseen characters to handle dense medical terms.
The first phase covers 500 billion tokens across the full FineMed-fr and FineMed-rephrased-fr datasets, chunking long documents hierarchically to fit the context window. The second phase consists of a 200-billion-token annealing stage restricted to biomedical and clinical subdomains to align representations closely with downstream clinical applications.
Key facts
- Architecture
- RoBERTa
- Parameters
- 111M (85M backbone, 26M embeddings)
- Context length
- 512 tokens
- Vocabulary size
- 32,768 tokens
- Primary language
- French
- Training data
- FineMed-fr and FineMed-rephrased-fr
What you can use it for
- Clinical Named Entity Recognition
- Fine-tuning to identify medical entities such as pathologies, drugs, examinations, and biometry, as well as qualifiers like negation, dates, and family relationships in clinical French text.
- Medical Document and Specialty Classification
- Classifying biomedical documents into medical specialties or diagnostic categories, as evaluated on benchmarks such as MorFITT and DiaMed.
- Biomedical Semantic Retrieval
- Producing dense vector representations for medical text retrieval and indexing when paired with sentence embedding or late-interaction frameworks.
- Masked Language Modelling
- Predicting masked medical terms within French clinical narratives and biomedical literature to evaluate term likelihood.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
DoctoBERT-fr-base
Strengths and limits
Good at
- Achieves the top aggregate score on the DrBenchmark evaluation across seven biomedical and clinical tasks compared to French generalist, French medical, and English medical encoders.
- Employs a custom SentencePiece tokeniser trained on high-density medical text with single-digit splitting and byte fallback for reliable clinical terminology processing.
- Combines web-curated medical text with synthetically rephrased variants to enhance robustness across diverse linguistic registers and noisy clinical writing styles.
Watch out for
- Context length is restricted to 512 tokens, requiring longer documents to be chunked or analysed with longer-context architectures.
- The model is an encoder rather than a generative model and cannot draft summaries, free-form answers, or clinical notes.
- It is not a certified medical device and its outputs must not be used to drive clinical decision-making.
- Pretrained on public web, PDF, and encyclopaedic sources, which may introduce underlying domain-specific or corpus-level biases.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use DoctoBERT-fr-base commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run DoctoBERT-fr-base without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
Can this model generate clinical text or discharge summaries?
No. DoctoBERT-fr-base is a masked encoder designed for representation learning, sequence classification, and entity recognition, not text generation.
How does the model handle long clinical documents?
Input length is capped at 512 tokens. Documents exceeding this limit must be split into sections before processing, or handled using a dedicated long-context variant such as DoctoModernBERT-fr-base.
What tasks was the model evaluated on?
It was evaluated on DrBenchmark across seven tasks: QUAERO (EMEA and MEDLINE), E3C (clinical and temporal NER), DEFT-2021, MorFITT specialty classification, and DiaMed diagnostic classification, as well as an in-house real-world French clinical NER dataset.
What languages does the model support?
The model is specifically trained on French biomedical corpora and is designed for French-language biomedical and clinical text.
How to cite it
Huang, B., Sun, J., Bouchacourt, D., Barascud, N., & Fogel, F. (2026). Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining. arXiv:2606.22079.
Written from DoctoBERT-fr-base's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.