nf-core pipeline · ddbj, download, ena
nf-core/fetchngs
Pipeline to fetch metadata and raw FastQ files from public repositories for downstream bioinformatics analyses.
What nf-core/fetchngs is
nf-core/fetchngs is a bioinformatics pipeline developed to retrieve metadata and raw FastQ sequencing data from public archives. It accepts accession identifiers from the Sequence Read Archive (SRA), European Nucleotide Archive (ENA), DNA Data Bank of Japan (DDBJ), and Gene Expression Omnibus (GEO).
The pipeline was originally developed by Harshil Patel and Jose Espinosa-Carrasco, with support for downloads via sra-tools contributed by Moritz E. Beber. It automates the extraction of experiment-level metadata and raw sequencing reads, resolving a common bottleneck when collecting published datasets for secondary analysis.
What the pipeline does
The pipeline ingests a text file listing database accession identifiers one per line. It maps these identifiers to experiment-level accessions compatible with the ENA API and queries the API to collect extensive study metadata. Sequencing reads are then acquired using one of three download methods.
If direct FastQ links exist within the ENA API, files are downloaded in parallel using wget with md5sum integrity validation by default, or optionally via Aspera CLI. When direct links are unavailable, the workflow uses sra-tools to download .sra archives and convert them into FastQ format. Finally, it compiles all gathered metadata alongside the local FastQ file paths into a single structured samplesheet.
Key facts
- Input format
- Single CSV or text file containing database accessions one per line
- Output files
- Raw FastQ files and a collated metadata samplesheet
- Supported databases
- SRA, ENA, DDBJ, and GEO
- Download methods
- FTP via wget (default), Aspera CLI, and sra-tools
- Downstream pipeline support
- nf-core/ampliseq, atacseq, mag, metatdenovo, rnaseq, sarek, taxprofiler, and viralrecon
- Latest release
- 1.12.0 (2024-02-29)
What you can use it for
- Public dataset retrieval
- Batch downloading sequencing reads and associated metadata from public archives using a list of SRA, ENA, DDBJ, or GEO accessions.
- Downstream pipeline preparation
- Generating structured samplesheets formatted specifically to pass downloaded data directly into analysis pipelines such as nf-core/rnaseq, nf-core/sarek, or nf-core/mag.
- Verified sequencing data acquisition
- Ensuring data integrity for public sequencing runs via automatic md5sum checksum validation during FTP or Aspera transfers.
Run it here
The research agent runs nf-core/fetchngs with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
Strengths and limits
Good at
- Validates file integrity automatically using md5sum checksums when downloading via FTP or Aspera.
- Formats output samplesheets for direct, out-of-the-box compatibility with multiple downstream nf-core pipelines.
- Handles accessions lacking direct FTP download links by falling back on sra-tools to extract FastQ files from SRA archives.
Watch out for
- Pipeline parameters cannot be supplied via custom configuration files using the Nextflow -c flag and must be passed using the command line or parameter files.
- Processing runs without direct ENA links requires downloading SRA files and running format conversion through sra-tools.
Questions
Can I run nf-core/fetchngs without my own cluster?
The research agent runs nf-core/fetchngs with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
What licence is nf-core/fetchngs under?
MIT, like every nf-core pipeline.
Which database accessions are supported by the pipeline?
The pipeline supports accession identifiers from SRA, ENA, DDBJ, and GEO provided in a single file one per line.
How does the pipeline transfer sequencing files?
By default, the pipeline uses wget over FTP to download FastQ files in parallel and performs an md5sum check. Users can alternatively force downloads via Aspera CLI or use sra-tools to download .sra files and convert them to FastQ.
Which nf-core pipelines can directly use the generated samplesheet?
The samplesheet can be formatted for nf-core/ampliseq, nf-core/atacseq, nf-core/mag, nf-core/metatdenovo, nf-core/rnaseq, nf-core/sarek, nf-core/taxprofiler, and the Illumina mode of nf-core/viralrecon.
How to cite it
10.5281/zenodo.5070524
Written from nf-core/fetchngs's own README. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.