Problems getting two output files in Nextflow

Question

Hello all!

I´m trying to write a small Nextflow pipeline that runs vcftools comands in 300 vcf´s. The pipe takes four inputs: vcf, pop1, pop2 and a .txt file, and would have to generate two outputs: a .log.weir.fst and a .log.log file. When i run the pipeline, it only gives the .log.weir.fst files but not the .log files.

Here´s my process definition:

process fst_calculation {

publishDir "${results_dir}/fst_results_pop1_pop2/", mode:"copy"

    input:
    file vcf
    file pop_1
    file pop_2
    file mart

    output:
    path "*.log.*"

    """
    while read linea
            do
                echo "[DEBUG] working in line: \$linea"
                inicio=\$(echo "\$linea" | cut -f3)
                final=\$(echo "\$linea" | cut -f4)
                cromosoma=\$(echo "\$linea" | cut -f1)
                segmento=\$(echo "\$linea" | cut -f5)
                vcftools --vcf ${vcf} \
                     --weir-fst-pop ${pop_1} \
                     --weir-fst-pop ${pop_2} \
                     --out \$inicio.log --chr \$cromosoma \
                     --from-bp \$inicio --to-bp \$final
            done < ${mart}
    """
}

And here´s the workflow of my process

/* Load files  into channel*/
pop_1 = Channel.fromPath("${params.fst_path}/pop_1")
pop_2 = Channel.fromPath("${params.fst_path}/pop_2")
vcf = Channel.fromPath("${params.fst_path}/*.vcf")
mart = Channel.fromPath("${params.fst_path}/*.txt")

/* Import modules
*/
 include {
   fst_calculation } from './nf_modules/modules.nf'

 /*
  * main pipeline logic
  */

 workflow  {
     p1 = fst_calculation(vcf, pop_1, pop_2, mart)
     p1.view()
 }

When i check the work directory of the pipeline, I can see that the pipe only generates the .log.weir.fst. To verify if my code was wrong, i ran "bash .command.sh" in the working directory and this actually generates the two output files. So, is there a reason for not getting the two output files when i run the pipe?

I appreciate any help.

score 0 · Accepted Answer · answered Oct 27 '21 at 15:08

Note that bash .command.sh and bash .command.run do different things. The latter is basically a wrapper around the former that sets up the environment and stages the declared input files, among other things. If running the latter produces the unusual behavior, you'll need to dig deeper.

It's not completely clear to me what the problem is here. My guess is that vcftools might behave differently when run non-interactively, such that it sends it's logging to STDERR. If that's the case, the logging will be captured in a file called .command.err. To instead send that to a file, you can just redirect STDERR in the usual way, untested:

while IFS=\$'\\t' read -r cromosoma null inicio final segmento ; do
    >&2 echo "[DEBUG] Working with: \${cromosoma}, \${inicio}, \${final}, \${segmento}"
    vcftools \\
         --vcf "${vcf}" \\
         --weir-fst-pop "${pop_1}" \\
         --weir-fst-pop "${pop_2}" \\
         --out "\${inicio}.log" \\
         --chr "\${cromosoma}" \\
         --from-bp "\${inicio}" \\
         --to-bp "\${final}" \\
         2> "\${cromosoma}.\${inicio}.\${final}.log.log"
done < "${mart}"

@fernanda-miron No worries at all! One more thing I thought of was to check your input channel declaration. Usually, but not always, process declarations will specify a [queue channel](https://www.nextflow.io/docs/latest/channel.html#queue-channel) and zero or more [value channels](https://www.nextflow.io/docs/latest/channel.html#value-channel). It may not be applicable here but this part of the documentation is well worth a read: [Understand how multiple input channels work](https://www.nextflow.io/docs/latest/process.html#understand-how-multiple-input-channels-work) — Steve, Oct 28 '21 at 10:57

Problems getting two output files in Nextflow

1 Answers1