4

I'm super confused about how to use GNU parallel to pass stdin to the job command.

I have what I imagined to be a really common use case. I have some process xxd that does something with stdin and outputs to stdout. I have some way to generate or get work from another standard stream, for exampleseq 3, and I can combine the two and make an impromptu power tool like so:

$ seq 3 | while read line; do echo $line | xxd; done
00000000: 310a                                     1.
00000000: 320a                                     2.
00000000: 330a                                     3.

Great. We can clearly see that each invocation of xxd gets one line, and a trailing newline is appended.

This is what piping to parallel does:

$ seq 3 | parallel --pipe --recend="\n" -L 1 xxd
...

00000000: 310a 320a 330a                           1.2.3.

parallel --pipe takes all of stdin and sends it to one invocation of xxd which confuses me because all the documented parameters and their defaults seem to contradict this behavior: --recend="\n" (the default) delimits jobs by newline, -L 1 (the default) sends a maximum of one line to the command.

Null separators have the same problem. They're also passed though verbatim:

seq 3 | tr '\n' '\0' | parallel --null --pipe xxd
...

00000000: 3100 3200 3300                           1.2.3.

An explanation for this behavior would be appreciated, especially since these parameters appear to apply specifically to the --pipe mode of parallel.

robinCTS
  • 4,407
ThorSummoner
  • 1,240

2 Answers2

8

You are SO close. -L sets the record size (in lines) but not how many records should be sent. This is controlled by -N. Default --recend is \n so that is not needed. Default -L is 1, so that is not needed.

seq 3 | parallel --pipe -N 1 xxd
Ole Tange
  • 5,099
1
  • --pipe sends stdin to job, implies --recend '\n'
  • -N1 send one captured input to one job
seq 3 | parallel --pipe -N1 xxd

Old Answer:

Warning: this may corrupt your data stream as echo is interpreting the payload. It may work for ascii and utf-8 encoded strings, but I wouldn't trust it generally!

Instead of pipe mode, echo (or similar) can convert args back to stout per-job

$ seq 3 | parallel "echo {} | xxd"
00000000: 310a                                     1.
00000000: 320a                                     2.
00000000: 330a                                     3.

ThorSummoner
  • 1,240