Putting it all together

In the end, a module consists of the following:

• main.nf contains the code for workflow and process definition. We duplicate ALL the parsing code (for CLI, input, ConfigMap, etc.) in order for a module to be effectively standalone.

• nextflow.config contains the ConfigMap for this specific module, scoped properly.

• Executables or scripts required to be on the $PATH for this module to run inside the container defined in nextflow.config.

In what follows, we will point to an example pipeline in viash_docs. This repository contains the source files needed to generate the DiFlow modules.

Creating a pipeline from these modules is now a matter of:

Generate the modules

Using viash, it’s easy to go from the component definitions under src/ to proper DiFlow modules:

viash ns build -p docker --setup
viash ns build -p nextflow

The first instruction builds the Docker containers needed for the pipeline to work. The second one builds the NextFlow/DiFlow modules.

Please note that viash allows you to also export to native or containerized binaries as well as run unit tests for the components. This, though, is covered elsewhere.

Pipeline main.nf

The pipeline logic is contained in main.nf. In order to use the modules defined using DiFlow, they have to be imported. This is the full main.nf file for the civ6_postgame pipeline:

nextflow.preview.dsl=2

import java.nio.file.Paths

include  plot_map       from  './target/nextflow/civ6_save_renderer/plot_map/main.nf'       params(params)
include  combine_plots  from  './target/nextflow/civ6_save_renderer/combine_plots/main.nf'  params(params)
include  convert_plot   from  './target/nextflow/civ6_save_renderer/convert_plot/main.nf'   params(params)
include  parse_header   from  './target/nextflow/civ6_save_renderer/parse_header/main.nf'   params(params)
include  parse_map      from  './target/nextflow/civ6_save_renderer/parse_map/main.nf'      params(params)
include  rename         from  './src/utils.nf'

workflow {

    if (params.debug == true)
        println(params)

    if (!params.containsKey("input") || params.input == "") {
        exit 1, "ERROR: Please provide a --input parameter pointing to .Civ6Save file(s)"
    }

    def input_ = Channel.fromPath(params.input)

    def listToTriplet = { it -> [ "all", it.collect{ a -> a[1] }, params ] }

    input_ \
        | map{ it -> [ it.baseName , it ] } \
        | map{ it -> [ it[0] , it[1], params ] } \
        | ( parse_header & parse_map ) \
        | join \
        | map{ id, parse_headerOut, params1, parse_mapOut, params2 ->
            [ id, [ "yaml" : parse_headerOut, "tsv": parse_mapOut ], params1 ] } \
        | plot_map \
        | convert_plot \
        | rename \
        | toSortedList{ a,b -> a[0] <=> b[0] }  \
        | map( listToTriplet ) \
        | combine_plots

}

Given the steps described above, we estimate that it’s possible to understand this pipeline.

Pipeline nextflow.config

This is the config file for the pipeline:

includeConfig 'target/nextflow/civ6_save_renderer/plot_map/nextflow.config'
includeConfig 'target/nextflow/civ6_save_renderer/combine_plots/nextflow.config'
includeConfig 'target/nextflow/civ6_save_renderer/convert_plot/nextflow.config'
includeConfig 'target/nextflow/civ6_save_renderer/parse_header/nextflow.config'
includeConfig 'target/nextflow/civ6_save_renderer/parse_map/nextflow.config'

docker {
  runOptions = "-i -v ${baseDir}:${baseDir}"
}

Running the pipeline

> nextflow run . \
  --input "data/*.Civ6Save" \
  --output "output/" \
  --combine_plots__framerate 1 \
 
N E X T F L O W  ~  version 20.10.0
Launching `./main.nf` [serene_mercator] - revision: 86da0cc3ec
executor >  local (26)
[2c/970402] process > parse_header:parse_header_process (AutoSave_0158) [100%] 5 of 5 ✔
[7d/c19cfa] process > parse_map:parse_map_process (AutoSave_0162)       [100%] 5 of 5 ✔
[06/4b19be] process > plot_map:plot_map_process (AutoSave_0160)         [100%] 5 of 5 ✔
[fc/3f219c] process > convert_plot:convert_plot_process (AutoSave_0162) [100%] 5 of 5 ✔
[f2/c24399] process > rename (AutoSave_0162)                            [100%] 5 of 5 ✔
[fb/43a707] process > combine_plots:combine_plots_process (all)         [100%] 1 of 1 ✔

Please note that we use an option --combine_plots__framerate 1. This points to an option of the combine_plots module that is called framerate. In other words, if a module defines an option (corresponding to a CLI option) it can be overridden from the CLI by using the convention <module_name>__<module option> <value>[^4].