Using zcat and gzip in linux

Question

I used zcat and gzip to merge and zip the files using:

zcat file1.fastq.gz file2.fastq.gz | gzip -c > file1.fastq.gz

Now, I have 0 bytes file1.fastq.gz

Please advise?

Answer 1

When you use the > redirection, the shell will open the named file (file1.fastq.gz) for writing and truncate it to zero length. (documentation)

Next, zcat will run, with file1.fastq.gz and file2.fastq.gz as input files .

At this point, zcat throws an error, because the input file (file1.fastq.gz) has no content - not even the GZip header - and the pipeline falls apart.

You may find that file1.fastq.gz is actually a few bytes in size (not zero), and this will be caused by gzip compressing and writing its null input.

$ echo "foo" | gzip > file1.gz
$ echo "bar" | gzip > file2.gz
$ zcat file1.gz file2.gz | gzip -c > file1.gz

gzip: file1.gz: unexpected end of file
$ stat -c '%s bytes' file1.gz
20 bytes

It's worth looking out for error messages like this.

---

There are a couple of viable solutions, but they depend on the size of your data.

In all situations, without some clever tricks, you will need to have enough space to store both the original and the new output file on disk at once.

Write to Another File

$ echo "foo" | gzip > file1.gz
$ echo "bar" | gzip > file2.gz
$ zcat file1.gz file2.gz | gzip -c > file_all.gz
$ gzip -d < file_all.gz
foo
bar

Use sponge

If the data isn't too large, then the sponge utility will handle this situation for you. All data read from stdin is "soaked up" before any data is written to the output file.

$ echo "foo" | gzip > file1.gz
$ echo "bar" | gzip > file2.gz
$ zcat file1.gz file2.gz | gzip -c | sponge file1.gz
$ gzip -d < file1.gz
foo
bar