nf-core pipeline ·
nf-core/datasync
A Nextflow pipeline for transferring files and directories across storage systems while validating their integrity.
What nf-core/datasync is
nf-core/datasync is a system operation and automation pipeline developed within the nf-core framework. Originally written by Alexander Peltzer, it automates the transfer of data between local filesystems and remote cloud storage systems, providing verification logs and reports to ensure data integrity.
Scientific research often involves moving large datasets across heterogeneous storage systems, such as synchronising sequencing runs or reference datasets between local computing nodes and cloud object storage. Incomplete transfers or silent corruption during data migration can undermine subsequent analyses. nf-core/datasync provides a structured workflow that validates sources against cryptographic checksum manifests, executes transfers using rclone, compares the copied data to the source, and compiles audit reports via MultiQC.
What the pipeline does
The pipeline processes each entry defined in an input CSV samplesheet across four sequential phases. First, it validates source files against provided MD5 or SHA-256 checksum manifests using the rclone checksum command. Second, it executes file transfers to designated targets with rclone copy. Third, it verifies the transferred files against the source data using rclone check to ensure consistency.
Throughout the process, the pipeline generates individual rclone status records. Finally, these individual metrics and transfer outcomes are collated into a unified MultiQC quality report.
Key facts
- Pipeline framework
- Nextflow (version 25.10.4 or later)
- Underlying transfer tool
- rclone
- Reporting tool
- MultiQC
- Input format
- CSV samplesheet specifying sample, input, output_path, checksum_md5, and checksum_sha
- Supported protocols
- Local paths, Amazon S3, S3-compatible storage, Azure Blob Storage
- Primary tested workflow
- S3-to-S3 bucket transfers
- Primary developer
- Alexander Peltzer
What you can use it for
- Cloud bucket synchronization
- Transferring datasets between Amazon S3 buckets or S3-compatible endpoints while documenting data completeness.
- Data ingestion to cloud storage
- Uploading local experimental data, such as sequencing runs or reference genomes, to remote object storage with pre- and post-transfer checksum validation.
- Transfer dry runs
- Previewing and auditing transfer commands using dry-run flags to evaluate configuration parameters and destination states without writing data.
Run it here
The research agent runs nf-core/datasync with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
Strengths and limits
Good at
- Automates both pre-transfer checksum verification (MD5 and SHA-256) and post-transfer comparison checks.
- Summarises transfer outcomes and rclone status files into a consolidated MultiQC report.
- Supports multiple storage backends, including local filesystems, Amazon S3, S3-compatible storage, and Azure Blob Storage.
- Provides a dry-run mode to inspect transfer operations without transferring data.
Watch out for
- HTTP and HTTPS URLs are not supported as input or output paths in the samplesheet.
- The primary tested configuration is transfer between S3 buckets; other providers require manual configuration and validation.
- Requires Nextflow version 25.10.4 or later.
Questions
Can I run nf-core/datasync without my own cluster?
The research agent runs nf-core/datasync with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
What licence is nf-core/datasync under?
MIT, like every nf-core pipeline.
What path formats are supported in the samplesheet?
The samplesheet accepts local filesystem paths and standard object-storage URIs, such as s3://bucket/path. It does not accept rclone-specific remote:path syntax, nor does it support HTTP or HTTPS URLs.
How are cloud storage credentials supplied?
When accessing endpoints that require credentials or custom provider settings, users must provide an rclone configuration file using the --rclone_config parameter.
What happens when running the pipeline in dry-run mode?
Enabling --rclone_dry_run simulates the copy operations without actually transferring files. Any post-copy comparison steps then report against whatever files already exist at the destination.
How to cite it
10.5281/zenodo.23104575
Written from nf-core/datasync's own README. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.