Snakemake wildcards II
In this video, I will show you how wildcards propagate within rules from the output to the inputs and shell commands. Let's start with a simple rule named copya1, which takes the file a, makes a copy, and renames it A1.
rule copya1:
input: "a.txt"
output: "a1.txt"
shell: "cp {input} {output}"
When we launch the pipeline, the rule is run, and the file a1 is generated. Now let's try to use wildcards for this rule. To do this, we define variable x for the output using curly brackets.
rule copya1:
input: "a.txt"
output: "{x}.txt"
shell: "cp {input} {output}"
When we launch the pipeline, of course, it crashes, because the output is not defined. Here snakemake patiently shares a bit of documentation and suggests using the rule all.
WorkflowError:
Target rules may not contain wildcards. Please specify concrete files or a rule without wildcards at the command line, or have a rule without wildcards at the very top of your workflow (e.g. the typical "rule all" which just collects all results you want to generate in the end).
So let's use the rule all, and specify a1 as our target file. If you are not sure how rule all works, please check the rule all video of the series. You can go to the video by clicking on the link on the screen.
rule all:
input: "a1.txt"
Now when we launch the pipeline, everything works, the file a1 is created, and the wildcard x is assigned a1. Let's see what happens if we try to specify the wildcard in the input. We will replace, in this case, the a with the x in curly brackets. When we launch the pipeline, it does not work, of course, because we are left with a cyclic dependency since we are giving the same file as input and output.
CyclicGraphException in rule copya1 in file /Users/Marcos/Desktop/snakemake/Snakefile, line 4:
Cyclic dependency on rule copya1.
So we need to be more careful in propagating the wildcard, and we have to write it in such a way that only the shared part of the string between the input and the output is placed between curly brackets. In this case, the variable is just a in the output, so we need to put the 1 after the wildcard.
rule copya1:
input: "{x}.txt"
output: "{x}1.txt"
shell: "cp {input} {output}"
Now the pipeline launches, the rules are run, and the a1 file is generated. In the process, the wildcard x has been assigned the value a and has been correctly replaced in the shell command.
Now let's try to propagate the wildcard to the Shell command by putting the wildcard between curly brackets. In this command, I will ask the shell to print the wildcard after making the file copy.
shell: "cp {input} {output} | echo {x}"
When we launch the pipeline, we get a NameError. And we are asked if we meant wildcards.x, which I suppose we did.
RuleException in rule copya1 in file /Users/Marcos/Desktop/snakemake/Snakefile, line 6:
NameError: The name 'x' is unknown in this context. Did you mean 'wildcards.x'?, when formatting the following:
cp {input} {output} | echo {x}
So let's go ahead and add wildcards before the x.
shell: "cp {input} {output} | echo file {wildcards.x}"
So now, when we launch the pipeline, the wildcard is printed as requested. And we can see it is used to generate the shell command.
OK, now let's try to propagate the wildcard across rules and through the input as well. We write the rule copya2 that makes a copy of file a1 and renames it a2. We make sure that the output of the copya1 matches the input of copya2.
rule copya2:
input: "{x}1.txt"
output: "{x}2.txt"
shell: "cp {input} {output}"
When launch the pipeline, all rules are run, and both files are generated. In every case, the wildcard x has been assigned the value a. We can see the DAG of the run with the following command.
snakemake --dag | dot -Tpdf x x.pdf
When we open the pdf, we see that copya1 runs first, copya2 next, and rule all last. When rule copya1 is run, the wildcard x is assigned the value a. And then gets propagated to rule copya2 without being shown in the DAG. Let's see what happens if we name the wildcards differently, by renaming the variables in copya2 y.
rule copya2:
input: "{y}1.txt"
output: "{y}2.txt"
shell: "cp {input} {output}"
And we get a similar result, but now the new y wildcards are specified in rule copya2.
WildcardError in file /Users/Marcos/Desktop/snakemake/Snakefile, line 11:
Wildcards in input files cannot be determined from output files:
'y'
So now, in addition to, propagating wildcards across rules, we can have multiple branches of the pipeline propagating different wildcards in parallel. For example, let's request the files b2, c2, and d2 in addition to a2. For this, we need to provide files b, c, and d as input as well.
rule all:
input: "a2.txt", "b2.txt", "c2.txt", "d2.txt"
So when we launch the pipeline, all requested files are generated. We can see that the different branches are run in parallel, with the same rules being assigned different wildcards for each branch.