Code Pretraining · ADSKAILab
codeparrot_megatron
A Megatron-formatted release of the CodeParrot dataset for large-scale code language-model pretraining.
What codeparrot_megatron is
codeparrot_megatron is a code pretraining dataset curated by Autodesk AI Lab (ADSKAILab). It provides the CodeParrot corpus prepared in the Megatron format to support large-scale pretraining runs.
The dataset is designed for research in engineering and scientific reasoning, providing a structured foundation for training language models to understand and generate programming code.
How it was made
The dataset arranges the CodeParrot source code corpus into the data structures and formats required by the Megatron framework, enabling distributed model training across multi-node infrastructure.
Key facts
- Publisher
- Autodesk AI Lab (ADSKAILab)
- Category
- Code Pretraining
- Data format
- Megatron
- Fields
- Engineering, scientific-reasoning
- Publication date
- 29 May 2025
What you can use it for
- Large-Scale Code Pretraining
- Supplying tokenised code data directly into Megatron-based training infrastructure for pretraining language models.
- Engineering Code Modelling
- Training specialised models for code analysis, program synthesis, and engineering workflow automation.
- Scientific-Reasoning Research
- Investigating how training on software repositories improves machine reasoning capabilities on scientific problems.
Explore the data
No licence declared
The authors have not published a licence, which means no permission to redistribute the data has been granted to anyone. The rows are not served here; read them on the original dataset page.Strengths and limits
Good at
- Supplied ready for use within Megatron training pipelines without requiring bespoke format conversion.
- Configured specifically for large-scale distributed pretraining runs.
Watch out for
- Requires tooling compatible with the Megatron data format.
- Source metadata does not specify language breakdown, filter criteria, or overall token count.
Questions
Can I look inside codeparrot_megatron without downloading it?
Not here: it declares no licence, so no permission to redistribute its rows has been granted. Its description and statistics are still shown.
What licence is codeparrot_megatron under?
Its authors have not declared a licence, which means no permission to reuse or redistribute it has been granted.
What is the difference between this dataset and standard CodeParrot?
This release is specifically formatted for use with the Megatron training framework for large-scale language-model pretraining experiments.
Who developed this release?
It was developed and published by Autodesk AI Lab (ADSKAILab).
Which research areas does this dataset address?
It is focused on code pretraining for engineering and scientific-reasoning applications.
Written from codeparrot_megatron's own dataset card. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.