finemed-fr

What finemed-fr is

FineMed-fr is a large corpus of French medical text designed for pretraining language models, consisting of 21,064,129 documents and 19.21 billion words. Published by doctolib-lab, it addresses the scarcity, limited scale, and narrow stylistic diversity seen in traditional curated medical corpora by compiling French-language health writing directly from open-web sources.

The dataset is derived from three parent corpora: FineWeb-2, FinePDFs, and FineWiki. To support downstream research and flexible data filtering, every document in the corpus is annotated across three quality dimensions: 15 clinical and biomedical subdomains, an educational quality score adapted from FineWeb-Edu, and medical-term density based on extracted terminology spans.

FineMed-fr serves as the pretraining data source for medical language models such as DoctoBERT and DoctoModernBERT. It is released without filtering, enabling researchers to tune thresholds and select document subsets suited to tasks such as fill-mask representation learning or text generation.

How it was made

The corpus builds on upstream curation from FineWeb-2, FinePDFs, and FineWiki, including language identification, heuristic quality filtering, and deduplication. To isolate medical content from broader web text, a multilingual DeBERTa-v3 domain classifier covering 26 domains evaluates the first 512 tokens of each document, retaining only texts whose top prediction is Health. This prefiltering step retains approximately 5.3% of FineWeb-2, 7.7% of FinePDFs, and 1.5% of FineWiki documents.

Retained documents are processed through three lightweight annotator models trained via two-stage knowledge distillation from larger language models. A subdomain classifier evaluates the text and URL to assign one of 15 medical subdomains. An educational scorer assigns an educational-quality rating from 0 to 5 based on a medical adaptation of the FineWeb-Edu rubric. A medical-entity extractor labels spans across eight taxonomic classes derived from UMLS (disease, drug, body part, medical procedure, molecular marker, clinical device, vital function, and living beings), and calculates medical entity density from the proportion of characters within these spans over the middle 512-token window.

Key facts

What you can use it for

Medical language model pretraining

Pretrain French clinical or biomedical encoder and decoder models, providing general foundation representations as used in DoctoBERT and DoctoModernBERT.

Domain-specific quality filtering

Filter large-scale web text using educational quality scores and medical-entity density thresholds to construct high-signal biomedical training sets.

Subdomain-targeted fine-tuning

Isolate specialized subsets across 15 distinct health subdomains, such as clinical guidelines, clinical vignettes, pharmacology trials, or public health policy, to train task-specific models.

Explore the data

The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.

Strengths and limits

Questions

How to cite it

Huang, Bofeng, Sun, Jacques, Bouchacourt, Diane, Barascud, Nicolas, and Fogel, Fajwel. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining. arXiv:2606.22079, 2026.