Nextflow: Scalable, Shareable & Reproducible workflows across Clouds & Clusters
Nextflow: Scalable, Shareable & Reproducible workflows across Clouds & Clusters (Radoslav Suchecki).
Slides are available at https://csiro-crop-informatics.github...
*Please note: this webinar is a re-recording of an original live event that experienced technical difficulties.
QUESTION:
Are there any example projects demonstrating Nextflow connected to NCRIS / CSIRO infrastructure?
ANSWER
Assuming you are running on CSIRO's pearcey the example workflow presented in the webinar https://github.com/csiro-crop-informa... should work. On an interactive node/session 'module load' the latest singularity module then you can run the example workflow using -profile slurm,singularity,singularitymodule. Feel free to contact me if you have any problems - in recent months running singularity with `automounts=true` on some HPC systems became problematic. I am using a workaround but a more permanent fix will be needed either in Singularity or Nextflow. For everyone else it you don't know how to run the example workflow on your HPC system, this might be a really good exercise - the pipeline logic is in place and so are some execution profiles defined to learn from. You can start by cloning the pipeline repo and adding another profile to nextflow.config - let me know if you get stuck.
QUESTION
I've played with NextFlow and a few other workflow engines in the past. One thing I could never figure out how do do with NextFlow was to decouple the wiring of the workflow from the task definitions, in order to have reusable task definitions (eg a reusable wrapper for a particular commandline program). Are you aware if this is possible in more recent versions (including splitting a workflow/tasks across multiple files, with imports/includes)?
ANSWER
I think there are two different things that we can consider here, one is the re-usable script block definition - what normally goes into the script block can be instead included under templates/ and used within a process definition you can think of it as a wrapper but things like directives/input/output blocks still have to be defined as normal. As for a reusable process definitions (tasks in Nextflow speak are instances of a process) this is being developed, follow this thread: https://github.com/nextflow-io/nextfl... - I have had a play with an early version and it looks really promising.
QUESTION
Can you set the max number of tasks allowed?
ANSWER
If you mean the number of potentially concurtently running tasks then the maxForks directive can be used for that (https://www.nextflow.io/docs/latest/p.... If this is about not processing all the samples but a subset you could, for example, conditionally use the .take() operator similarly to how it is done in the example workflow (https://github.com/csiro-crop-informa....
QUESTION
What does "reactive" mean?
ANSWER
According to NF authors the term is used to stress the asynchronous-declarative programming model used by Nextflow, that it is based on the dataflow paradigm on top of which some reactive programming primitive has been built. See https://en.wikipedia.org/wiki/Dataflow, https://en.wikipedia.org/wiki/Reactiv....
As I understand it, this reactivity along with the broader dataflow paradigm allow us to develop workflow logic without describing its control flow. We simply start with some input data channel(s) and apply processes and operators which either transform or redirect the data units flowing through the channels.
QUESTION
Is "io" just the domain name for nextflow? i.e. it is not needed as workflow file name extension.
ANSWER
It is just a top level domain used for Nextflow website. The workflow script can have any name but main.nf is the default - useful if you want to run a workflow directly from a repository via 'nextflow run user/repo' without re-defining the name in nextflow.config.
QUESTION
What info/data is used to generate the hashes for the directory names in the work directory?
ANSWER
In simple terms, a hash for a given task is computed from its inputs and the command script, here is the relevant code in all its glory: https://github.com/nextflow-io/nextfl...