nf-core pipeline · benchmark, small-variants, structural-variants
nf-core/variantbenchmarking
Benchmarks germline and somatic small variants, structural variants and copy number alterations against truth sets.
What nf-core/variantbenchmarking is
nf-core/variantbenchmarking is a bioinformatics analysis pipeline designed to assess and validate the performance of genomic variant callers. Originally developed by Kübra Narcı for the German Human Genome-Phenome Archive (GHGA), it compares query variant call format (VCF) files against gold standard reference datasets or against each other. It handles germline and somatic analyses across single-nucleotide variants (SNVs), insertions and deletions (indels), structural variants (SVs), and copy number variations (CNVs).
Benchmarking genomic variant calling requires rigorous standardization because different callers represent complex alleles, breakends, and indel alignments inconsistently. This pipeline resolves these discrepancies by normalizing, decomposing, and standardizing variants prior to assessment. Where validated reference standards such as Genome in a Bottle (GIAB) or SEQC2 are unavailable, the workflow can also construct an ensemble proxy truth set using consensus voting across multiple callers.
The pipeline automates downstream statistical summaries, caller concordance analyses, and classifications into true positives, false positives, and false negatives. It generates consolidated evaluation reports, enabling researchers to make objective comparisons between algorithms, library preparations, and analytical pipelines.
What the pipeline does
Input test VCFs and baseline truth VCFs undergo standardization and normalization. For small variants, bcftools splits multiallelic sites, deduplicates, and left-aligns alleles, while prepy provides normalization for germline hap.py workflows. Structural variants are reformatted with svync, standardized or decomposed to breakends (BND) using SVTK or RTG Tools svdecompose, and homogenised with variant-extractor. When genomic coordinates differ, Picard LiftoverVcf and UCSC liftOver can re-map VCFs and high-confidence BED intervals to matching reference genome assemblies. If an ensemble baseline is chosen, bcftools merge or SURVIVOR merge combines variant calls across callers, applying a majority-rule threshold to define proxy truth variants.
Following optional contig and quality filtering, evaluation is delegated to specialized benchmarking engines based on the variant category. Small variants are evaluated with hap.py, som.py, or RTG Tools vcfeval (using squash-ploidy for somatic data). Structural variants are assessed using Truvari, SVanalyzer, or RTG Tools bndeval, while copy number variations are benchmarked with Wittyer, Truvari, RTG Tools cnveval, or bedtools intersect for region overlaps. Concordance between multiple test callers can be evaluated with GATK4 Concordance. Finally, true positive, false positive, and false negative sets are merged to identify tool-specific and shared calls, producing summary metrics, plots, MultiQC output, and datavzrd visual reports.
Key facts
- Pipeline framework
- Nextflow DSL2
- Initial author
- Kübra Narcı (GHGA project)
- Input formats
- CSV samplesheet, VCF (.vcf, .vcf.gz), BED
- Supported variant types
- SNVs, indels, structural variants (SVs), copy number variations (CNVs)
- Analysis modes
- Germline and somatic
- Benchmarking engines
- hap.py, som.py, RTG Tools (vcfeval, bndeval, cnveval), Truvari, SVanalyzer, Wittyer, bedtools
- Coordinate lift-over tools
- Picard LiftoverVcf, UCSC liftOver
- Reporting outputs
- CSV summaries, R plots, datavzrd HTML reports, MultiQC
What you can use it for
- Small Variant Caller Validation
- Benchmarking germline or somatic SNV and indel callers against Genome in a Bottle (GIAB) or SEQC2 reference truth sets using hap.py, som.py, or RTG Tools vcfeval.
- Structural Variant and CNV Evaluation
- Evaluating the precision and recall of SV and copy number callers using Truvari, SVanalyzer, Wittyer, RTG Tools, or interval intersection with bedtools.
- Consensus Truth Set Creation
- Generating a proxy ground truth when no reference standard exists by applying a majority-rule threshold across multiple caller outputs with bcftools and SURVIVOR.
- Cross-Assembly and Inter-Caller Concordance
- Lifting variant coordinates across genome assemblies (such as GRCh37 to GRCh38) while measuring pairwise concordance between different pipelines using GATK4.
Run it here
The research agent runs nf-core/variantbenchmarking with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
Strengths and limits
Good at
- Normalizes complex variants, decomposes structural breakends, and left-aligns indels to prevent formatting discrepancies from biasing benchmark metrics.
- Supports multiple variant classes—SNVs, indels, structural variants, and copy number variations—within a single framework.
- Enables benchmarking against consensus truth sets generated from majority-rule caller voting when experimental gold standards are unavailable.
- Integrates coordinate liftover for both VCF and high-confidence BED intervals to resolve reference genome version mismatches.
- Aggregates true positive, false positive, and false negative sets across callers to clarify shared and caller-unique detection profiles.
Watch out for
- No somatic-specific benchmarking engine is implemented for structural variants or copy number variations.
- Wittyer does not support breakend (BND) structural variants, requiring them to be filtered out or transformed before CNV benchmarking.
- Conda profiles are unsupported for SVanalyzer, requiring container runtimes such as Docker or Singularity when running that tool.
Questions
Can I run nf-core/variantbenchmarking without my own cluster?
The research agent runs nf-core/variantbenchmarking with Nextflow on its own machine — up to 32 cores, 256 GiB of memory and 24 hours per run, which covers the test profile and real single-sample runs. It builds the samplesheet, checks it against the pipeline's schema and reads the results back. When a run is too big for one machine, it gives you the exact command for your own cluster instead.
What licence is nf-core/variantbenchmarking under?
MIT, like every nf-core pipeline.
Can the pipeline benchmark variant calls if there is no official gold standard dataset?
Yes. By setting an ensemble threshold greater than zero, the pipeline merges calls from multiple tools using bcftools or SURVIVOR and applies majority-rule voting to generate a proxy truth set.
What reference truth datasets are commonly used with this pipeline?
Genome in a Bottle (GIAB) truth sets are commonly used for germline benchmarks, while SEQC2 reference datasets are standard for somatic SNV, indel, and CNV benchmarks.
Does the pipeline support benchmarking against truth files on a different reference genome build?
Yes. The pipeline includes subworkflows utilizing Picard LiftoverVcf and UCSC liftOver to convert coordinates of query VCFs, truth VCFs, and high-confidence BED regions to matching assemblies.
Why does the pipeline standardize structural variants into breakends (BND)?
Standardizing SVs into breakends decomposes complex rearrangements and harmonizes disparate caller representations, allowing consistent comparison between query calls and baseline truth sets.
How to cite it
110.5281/zenodo.14916661
Written from nf-core/variantbenchmarking's own README. Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.