Snakemake cannot handle very long command line?

Question

This is a very strange problem. When my {input} specified in the rule section is a list of <200 files, snakemake worked all right. But when {input} has more than 500 files, snakemake just quitted with messages (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!). The complete log did not provide any error messages.

For the log, please see: https://github.com/snakemake/snakemake/files/5285271/2020-09-25T151835.613199.snakemake.log

The rule that worked is (NOTE the input is capped to 200 files):

rule combine_fastq:
    input:
        lambda wildcards: samples.loc[(wildcards.sample), ["fq"]].dropna()[0].split(',')[:200]
    output:
        "combined.fastq/{sample}.fastq.gz"
    group: "minion_assemble"
    shell:
        """
echo {input} >  {output}
        """

The rule that failed is:

rule combine_fastq:
    input:
        lambda wildcards: samples.loc[(wildcards.sample), ["fq"]].dropna()[0].split(',')
    output:
        "combined.fastq/{sample}.fastq.gz"
    group: "minion_assemble"
    shell:
        """
echo {input} >  {output}
        """

My question is also posted in GitHub: https://github.com/snakemake/snakemake/issues/643.

I think it might be related to this: https://stackoverflow.com/questions/19354870/bash-command-line-and-input-limit — Maarten-vd-Sande, Sep 26 '20 at 13:08
I think the command line length in shell is not a problem. I ran `getconf ARG_MAX ` and got `4611686018427387903`. — zhanxw, Oct 19 '20 at 05:13
@Maarten-vd-Sande I have not figured that. I wrote a different script to bypass this problem in snakemake. — zhanxw, Oct 28 '20 at 14:18

score 2 · Answer 1 · edited Oct 29 '20 at 15:53

I second Maarten's answer, with that many files you are running up against a shell limit; snakemake is just doing a poor job helping you identify the problem.

Based on the issue you reference, it seems like you are using cat to combine all of your files. Maybe following the answer here would help:

rule combine_fastq_list:
    input:
        lambda wildcards: samples.loc[(wildcards.sample), ["fq"]].dropna()[0].split(',')
    output:
        temp("{sample}.tmp.list")
    group: "minion_assemble"
    script:
        with open(output[0]) as out:
            out.write('\n'.join(input))

rule combine_fastq:
    input:
        temp("{sample}.tmp.list")
    output:
        'combined.fastq/{sample}.fastq.gz'
    group: "minion_assemble"
    shell:
        'cat {input} | '  # this is reading the list of files from the file
            'xargs zcat -f | '
            '...'

Hope it gets you on the right track.

edit

The first option executes your command separately for each input file. A different option that executes the command once for the whole list of input is:

rule combine_fastq:
    ...
    shell:
        """
        command $(< {input}) ...
        """

Thanks for the trick. I tried but unfortunately it did not work. — zhanxw, Oct 19 '20 at 05:14
This didn't work for me like this, but with using the input directly as argument: `$(< {input.files})`! — Maarten-vd-Sande, Oct 28 '20 at 19:57
@Maarten-vd-Sande can you edit my answer or post a new one? Not sure I know where to make that modification. — Troy Comi, Oct 29 '20 at 14:20

dariober · Answer 2 · 2021-03-02T09:45:51.283

For those landing here with similar questions (like Snakemake expand function alternative), snakemake 6 can handle long command lines. The following test fails on snakemake < 6 but succeeds on 6.0.0 on my Ubuntu machine:

rule all:
    input:
        'output.txt',

rule one:
    output:
        'output.txt',
    params:
        x= list(range(0, 1000000))
    shell:
        r"""
        echo {params.x} > {output}
        """

Snakemake cannot handle very long command line?

2 Answers2

Linked