Passing Python variables through Nextflow channels

Question

I have the following simplified task and get some errors when doing it.

I'm trying to get 4 letters from a string by Python and use that as a name of a new folder. My code is as follows:

#!/usr/bin/env nextflow

params.string="abcdefg"

process PYTHON{
    input:
    val string
    
    output:
    val substring
    
    """
    #!/usr/bin/env python
    
    substring="$string"[0:4]
    """
}

process CREAT_FOLDER{
    input:
    val name
    
    """
    mkdir $name
    """
}

workflow{
    name = PYTHON(params.string)
    CREAT_FOLDER(name)
}

The error shows up.

N E X T F L O W  ~  version 23.04.1
Launching `python.nf` [shrivelled_kay] DSL2 - revision: 0e01fd9b95
executor >  local (1)
executor >  local (1)
[33/e827c9] process > PYTHON       [100%] 1 of 1, failed: 1 ✘
[-        ] process > CREAT_FOLDER -
ERROR ~ Error executing process > 'PYTHON'

Caused by:
  Missing value declared as output parameter: substring

The variable "substring" clearly couldn't pass through channels. So how can I pass a Python variable into channels?

Steve · Answer 1 · 2023-05-10T10:48:36.883

The problem is that you need a globally scoped variable in order to declare it using the val qualifier. The usual solution is to just define these in your script block and then have Nextflow embed them in your Perl/Python/Bash script, for example:

params.string="abcdefg"

process PYTHON {

    debug true

    input:
    val string

    output:
    val substring

    script:
    substring = string.substring(0, 4)

    """
    #!/usr/bin/env python

    the_substring='${substring}'

    print(f'The substring: {the_substring}')
    """
}

process CREAT_FOLDER {

    debug true

    input:
    val name

    """
    mkdir -v "${name}"
    """
}

workflow {

    name = PYTHON(params.string)

    CREAT_FOLDER(name)
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [fervent_brazil] DSL2 - revision: 0676556654
executor >  local (2)
[3e/3f2501] process > PYTHON       [100%] 1 of 1 ✔
[5b/fbcccb] process > CREAT_FOLDER [100%] 1 of 1 ✔
The substring: abcd

mkdir: created directory 'abcd'

For completeness, the alternative solution is to of course write the variable to a file and then read the value from it in your workflow block. For example:

params.string="abcdefg"

process PYTHON {

    input:
    val string

    output:
    path 'result.txt'

    """
    #!/usr/bin/env python

    substring='${string}'[0:4]

    with open('result.txt', 'w') as out:
        out.write(substring)
    """
}

process CREAT_FOLDER {

    debug true

    input:
    val name

    """
    mkdir -v "${name}"
    """
}

workflow {

    name = PYTHON(params.string).map { it.text }

    CREAT_FOLDER(name)
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [nauseous_yalow] DSL2 - revision: 64ed37cd94
executor >  local (2)
[c6/ad2f57] process > PYTHON       [100%] 1 of 1 ✔
[ee/183f66] process > CREAT_FOLDER [100%] 1 of 1 ✔
mkdir: created directory 'abcd'

Passing Python variables through Nextflow channels

1 Answers1