Pretraining Corpus · allenai

peS2o

A multi-domain open-access scientific pretraining corpus of up to 40 million cleaned papers for language models.

What peS2o is

peS2o (Pretraining Efficiently on S2ORC) is a large text corpus curated by the Allen Institute for AI for pretraining science-aware language models. It is derived from open-access scientific literature in the Semantic Scholar Open Research Corpus (S2ORC) and the Semantic Scholar Academic Graph (S2AG), spanning disciplines including biology, chemistry, physics, computer science, and engineering.

Raw academic paper collections often contain broken text, non-textual elements, optical character recognition (OCR) artefacts, and non-English text, which hinder model training. peS2o addresses this by filtering, cleaning, and standardising document structures into plain text formatted specifically for language model pretraining.

The corpus provides researchers with cleaned full-text articles as well as title-and-abstract records across two major releases (V1 and V2), enabling large-scale text generation and masked language modelling tasks on scientific prose.

How it was made

peS2o is derived from two Semantic Scholar sources: full-text papers from S2ORC and paper titles and abstracts from S2AG. For full-text papers, Grobid extracts section headers and paragraphs while stripping figures, tables, references, and other non-textual components. Document language is determined using pycld3 on the first 2,000 characters of each paragraph, retaining only English documents published after 1969. Papers must contain at least 500 whitespace-separated words, at least five paragraphs, and an average unigram log probability of at least -20 computed using the 1T Web Ngram corpus. The most common word must be alphabetic and constitute less than 7.5% of the total words.

For title and abstract records (S2AG), pycld3 filters for English text, abstracts must have between 50 and 1,000 words, and the average unigram log probability must exceed -20. In peS2o V2, additional filtering removes abstracts containing OCR spacing defects, discarding records with more than four instances of separated alphabetical characters. Documents are split chronologically into training and validation sets based on publication dates.

Key facts

Corpus size (V2): 38.97 million documents (42.01 billion whitespace-separated tokens)
Corpus size (V1): 67.56 million documents (47.37 billion whitespace-separated tokens)
Knowledge cutoff: 2023-01-03
Primary language: English
Source datasets: S2ORC (full-text) and S2AG (titles and abstracts)
Document format: Dictionary containing added, created, id, source, text, and version fields
Intended tasks: Text generation and masked language modelling (fill-mask)

What you can use it for

Language model pretraining: Train foundational language models on cleaned, domain-specific academic literature spanning biology, chemistry, engineering, and physics.
Masked language modelling: Pretrain masked fill-mask transformers on large corpora of scientific abstracts and full-text prose.
Scientific metadata integration: Link text content to author, field, and citation metadata via Semantic Scholar Corpus IDs using the Semantic Scholar API.

Explore the data

No queryable preview

This dataset is not available in a queryable form — usually an unusual file format or a repository too large to convert. The original source may still be readable.

Strengths and limits

Good at

Provides tens of billions of cleaned words extracted from peer-reviewed and open-access scientific literature.
Applies rigorous quality filtering based on language identification, n-gram word likelihood, and structural length constraints.
Retains Semantic Scholar Corpus IDs for easy enrichment with external academic metadata.
Provides distinct train and validation splits based on publication date.

Watch out for

Excludes all non-textual information, including tables, figures, formulas, and bibliographic references.
Restricted entirely to English-language academic texts.

Questions

Can I look inside peS2o without downloading it?

Not here: it is not available in a queryable form. Its description and statistics are still shown.

What licence is peS2o under?

Its authors declare odc-by. Read the terms on the original page before reusing the data.

What is the difference between peS2o V1 and V2?

peS2o V2 implements stricter OCR error filtering on abstracts derived from S2AG, discarding abstracts with excessive spaced-out characters, resulting in a cleaner but smaller dataset of 38.97 million documents compared to 67.56 million in V1.

What fields of study are covered in peS2o?

The corpus spans a broad spectrum of scientific disciplines, including biology, chemistry, computer science, engineering, physics, materials science, mathematics, environmental science, psychology, geology, and social sciences.

How are the training and validation sets divided?

The training set contains documents published before 1 December 2022, while the validation set consists of documents published between 1 December 2022 and 3 January 2023.

Can I obtain paper metadata like authors or citations?

Yes, each entry includes a Semantic Scholar Corpus ID (`id`), which can be queried against the Semantic Scholar API to retrieve fields of study, authors, and citation details.

How to cite it

Soldaini, L. and Lo, K., 2023. peS2o (Pretraining Efficiently on S2ORC) Dataset. Allen Institute for AI. https://github.com/allenai/pes2o

Written from peS2o's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.