Multi-Modal Foundation · doctolib-lab
DoctoModernBERT-fr-base
A 149M-parameter French biomedical encoder with an 8,192-token context window for clinical NLP and token classification.
What DoctoModernBERT-fr-base is
DoctoModernBERT-fr-base is a dedicated French medical text encoder developed by doctolib-lab. Built upon the ModernBERT architecture, it is designed to process biomedical and clinical text natively in French, addressing common linguistic and domain-specific challenges found in real-world clinical documentation.
Clinical records in non-English languages often present high stylistic variation, noise, and complex terminology that standard generalist language models struggle to capture. By pretraining directly on curated French biomedical corpora with extended sequence lengths, the model offers researchers and clinicians a specialised foundation for downstream information extraction and classification tasks.
The model achieves the highest F1 score on a real-world clinical named entity recognition (NER) evaluation covering 12 entity types and 9 qualifiers, while ranking among top-performing French models across academic evaluations on DrBenchmark.
How it works
The model relies on a ModernBERT architecture with 149 million parameters (110 million backbone and 39 million embeddings), 22 layers, 12 attention heads, a hidden size of 768, an intermediate size of 1152, and GeGLU activation functions. It uses a SentencePiece BPE tokenizer with a 50,368-token vocabulary that splits digits into single tokens and incorporates byte fallback for unseen characters. The context window supports sequences up to 8,192 tokens.
Pretraining was conducted from scratch over three sequential phases totalling 240 billion tokens using FineMed-fr and FineMed-rephrased-fr, datasets derived from FineWeb-2, FinePDFs, and FineWiki filtered by educational quality and medical-term density. Phase 1 trained on 200 billion tokens via masked language modelling with a 1,024-token context, hierarchically chunking long documents. Phase 2 performed context extension to 8,192 tokens on 20 billion tokens upsampled toward longer texts. Phase 3 conducted annealing on 20 billion tokens focused solely on biomedical and clinical subdomains.
Key facts
- Architecture
- ModernBERT
- Parameters
- 149M (110M backbone, 39M embeddings)
- Layers
- 22
- Attention heads
- 12
- Hidden size
- 768
- Intermediate size
- 1152
- MLP activation
- GeGLU
- Context window
- 8,192 tokens
- Vocabulary size
- 50,368
- Primary language
- French
- Training data
- FineMed-fr and FineMed-rephrased-fr (240B tokens total)
What you can use it for
- Clinical Named Entity Recognition
- Fine-tune token classification heads to extract medical concepts such as pathologies, drugs, examinations, and biometrics, alongside qualifiers including negation and dates, from French clinical text.
- Long-Document Medical Classification
- Classify extended medical narratives, such as hospital reports, biomedical specialty texts (e.g. MorFITT), or diagnostic classifications (e.g. DiaMed), leveraging the full 8,192-token context window.
- Masked Language Modelling and Probing
- Perform fill-mask inference on French medical sequences to inspect term probabilities or analyse masked token contextual representations.
- Biomedical Text Retrieval and Embeddings
- Adapt the encoder with Sentence Transformers or PyLate to produce dense representations for French medical document search and clinical retrieval tasks.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
DoctoModernBERT-fr-base
Strengths and limits
Good at
- Supports long contexts natively up to 8,192 tokens, permitting the analysis of extended clinical records without premature truncation.
- Trained on curated, entity-dense corpora with LLM-rephrased variations designed to build robustness against noisy and irregular clinical text styles.
- Offers compatibility with FlashAttention-2 for faster and memory-efficient processing of long text sequences.
Watch out for
- Operates strictly as an encoder model and cannot generate free-form text or summaries.
- Is not a medical device and its outputs must not be used to drive clinical decisions.
- Reflects the biases, omissions, and demographic gaps present in open web, PDF, and encyclopedic pretraining data.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use DoctoModernBERT-fr-base commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run DoctoModernBERT-fr-base without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
Can this model generate text or conversational answers?
No. DoctoModernBERT-fr-base is an encoder model designed for representation, masked language modelling, and fine-tuning on discriminative tasks such as classification and entity extraction.
What context length does the model support?
The model supports context lengths up to 8,192 tokens, following a dedicated context-extension pretraining stage.
How should long input sequences be processed efficiently?
Its authors recommend loading the model with FlashAttention-2 enabled for faster computation and lower memory consumption on long texts.
What tasks was the model evaluated on?
It was evaluated on a proprietary real-world clinical NER benchmark and on DrBenchmark, which includes QUAERO (EMEA and MEDLINE), E3C clinical and temporal NER, DEFT-2021, MorFITT specialty classification, and DiaMed diagnostic classification.
How to cite it
Huang, Bofeng and Sun, Jacques and Bouchacourt, Diane and Barascud, Nicolas and Fogel, Fajwel. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining. arXiv:2606.22079, 2026.
Written from DoctoModernBERT-fr-base's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.