Workflows · Software
Nextflow
A workflow system for creating scalable, portable, and reproducible scientific data-analysis pipelines.
What Nextflow is
Nextflow is an open-source workflow management system created by Seqera. It is designed to construct and execute computational data-analysis pipelines that remain reproducible, portable, and scalable across heterogeneous environments. It also serves as the execution engine powering community pipelines in the nf-core project.
Scientific analysis often involves chaining multiple tools, scripts, and software environments together, which traditionally introduces difficulties when moving work from local development to institutional clusters or cloud environments. Nextflow addresses this by providing an abstraction layer between pipeline logic and execution platforms, allowing users to write computational pipelines without modifying code between different computing targets.
By adopting a dataflow programming paradigm, Nextflow simplifies parallel and distributed processing. It enables researchers to integrate existing code and command-line tools into cohesive workflows, tracking intermediate data to safeguard large-scale analyses against execution failures.
How it works
Nextflow is built on Groovy and GPars and extends the Unix pipes model using a domain-specific language (DSL). In this dataflow model, parallelisation is defined implicitly: individual computational tasks (processes) declare their input and output channels. When data becomes available in an input channel, the process executes, allowing tasks to run concurrently and scale up or scale out automatically across compute infrastructure.
The engine decouples the definition of tasks from their runtime execution through pluggable executors. It integrates with container runtimes and package managers to isolate software environments. Nextflow also performs continuous checkpointing by automatically tracking all intermediate files generated during execution, enabling interrupted pipelines to resume directly from the last successfully completed step.
Key facts
- Programming foundation
- Groovy and GPars
- Supported container runtimes
- Docker, Podman, Singularity
- Supported package managers
- Conda, Spack
- Supported HPC schedulers
- SLURM, GridEngine, LSF, PBS, Moab, HTCondor
- Supported cloud platforms
- AWS Batch, Azure Batch, Google Cloud Batch, Kubernetes
What you can use it for
- RNA sequencing analysis
- Executing pipelines such as nf-core/rnaseq that coordinate tools including STAR, RSEM, HISAT2, or Salmon to generate gene and isoform counts alongside quality control metrics.
- Germline and somatic variant detection
- Running variant-calling workflows such as nf-core/sarek to process whole-genome or targeted sequencing data through pre-processing, variant calling, and annotation.
- Sequence similarity searching
- Distributing large-scale sequence comparison workloads across multiple nodes using distributed BLAST pipelines.
- Machine learning pipelines
- Structuring multi-step data preparation, training, and evaluation workflows that leverage parallel execution.
Run it here
Ask the research agent and it runs Nextflow as a job on a dedicated machine, then reads the results back to you. Nothing to install, and it keeps running if you close the page.
Try asking
“Run the nf-core/rnaseq test profile and show me the MultiQC report.”
Strengths and limits
Good at
- Separates pipeline logic from deployment configurations so that identical code runs without modification across local machines, HPC clusters, and cloud batch services.
- Maintains continuous checkpoints of all intermediate results, allowing failed or interrupted runs to be resumed from the last completed process.
- Derives concurrency automatically from process input and output declarations using a dataflow model.
- Accommodates polyglot workflows by allowing existing scripts and tools across multiple scripting languages to be reused directly.
Watch out for
- Relies on external package managers or container technologies such as Conda, Docker, or Singularity to handle software dependencies.
- Depends on external batch schedulers or cloud provider services to orchestrate distributed compute infrastructure.
Questions
Can I run Nextflow online without installing it?
Ask the research agent and it runs Nextflow as a job on a dedicated machine, then reads the results back to you. Nothing to install, and it keeps running if you close the page.
What licence is Nextflow under?
Apache-2.0. The terms are published at https://github.com/nextflow-io/nextflow.
How does Nextflow handle task parallelisation?
Parallelisation is handled implicitly via its dataflow programming model. Processes run automatically as soon as their declared input data channels receive inputs, scaling across available compute cores or distributed nodes without explicit concurrency programming.
What happens if a pipeline run fails or is stopped?
Nextflow tracks all intermediate results through continuous checkpointing. If an execution stops for any reason, it can be resumed from the last successfully completed task rather than starting from the beginning.
Which batch schedulers and cloud environments can execute Nextflow pipelines?
It provides built-in executors for local execution, HPC schedulers including SLURM, GridEngine, LSF, PBS, Moab, and HTCondor, as well as Kubernetes, AWS Batch, Azure Batch, and Google Cloud Batch.
How does Nextflow isolate tool dependencies?
Workflows manage dependencies by integrating with container systems like Docker, Podman, and Singularity, or environment managers such as Conda and Spack.
How to cite it
P. Di Tommaso, et al. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 316–319 (2017) doi:10.1038/nbt.3820
Written from Nextflow's own README and project page. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.