Why does this not work? "ls *.txt | xargs cat > all.txt" (all files into single txt document)

Question

Why does this not work?

ls *.txt | xargs cat > all.txt

(I want to join the contents of all text files into a single 'all.txt' file.) find with -exec should also work, but I would really like to understand the xargs syntax.

Thanks

Answer 1

ls *.txt | xargs cat >> all.txt

might work a bit better, since it would append to all.txt instead of creating it again after each file.

By the way, cat *.txt >all.txt would also work. :-)

Answer 2

If some of your file names contain ', " or space xargs will fail because of the separator problem

In general never run xargs without -0 as it will come back and bite you some day.

Consider using GNU Parallel instead:

ls *.txt | parallel cat > tmp/all.txt

or if you prefer:

ls *.txt | parallel cat >> tmp/all.txt

Learn more about GNU Parallel http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 3

all.txt is a file in the same directory, so cat gets confused when it wants to write from the same file to the same file.

On the other hand:

ls *.txt | xargs cat > tmp/all.txt

This will read from textfiles in your current directory into the all.txt in a subdirectory (not included with *.txt).

Answer 4

You could also come across a command line length limitation. Part of the reason for using xargs is that it splits up the input into safe command-line-sized chunks. So, imagine a situation in which you have hundreds of thousands of .txt files in the directory. ls *.txt will fail. You would need to do

ls | grep .txt$ |xargs cat > /some/other/path/all.txt

.txt$ in this case is a regular expression matching everything that ends in .txt (so it's not exactly like *.txt, since if you have a file called atxt, then *.txt would not match it, but the regular expression would.)

The use of another path is because, as other answers have pointed out, all.txt is matched by the pattern *.txt so there would be a conflict between input and output.

Note that if you have any files with ' in their names (and this may be the cause of the unmatched single quote error), you would want to do

ls | grep --null .txt$ | xargs -0 cat > /some/other/path/all.txt

The --null option tells grep to use output separated by a \0 (aka null) character instead of the default newline, and the -0 option to `xargs tells it to expect its input in the same format. This would work even if you had file names with newlines in them.