6
seq 100 | ... wc -l

What can fill in for ... so that that wc -l prints 10, ten times? A clunky way I can think of is:

seq 100 | xargs -L10 bash -c '(for i; do echo $i; done) | wc -l' bash

Wondering if there's a better and idiomatic way to do the same.

Jeenu
  • 336

2 Answers2

7

GNU split

Use GNU split with its --filter option:

seq 100 | split -l 10 --filter='wc -l'

Note the value for --filter= is interpreted as shell code, so you can directly build more complex logic there; and split sets a variable named FILE for you to use. Example:

seq 100 | split -d -l 10 --filter='printf "%s: " "$FILE"; wc -l' - 'chunk '

GNU parallel

While split runs filters sequentially, GNU parallel can start many jobs in parallel. With seq 100 and wc -l this is overkill, but for more complicated jobs parallel makes sense.

seq 100 | parallel --pipe -n 10 --keep-order --group -- wc -l
Kamil Maciorowski
  • 81,893
0

https://superuser.com/a/1903026/41337 is a good answer.

If the input is a file, this will be faster:

parallel --pipe-part -a bigfile --block -1 wc -l

It will run one wc -l per CPU core and dynamically split bigfile into one chunk per wc -l.

It can process > 1GB/s per CPU core, so your disk (or wc -l) will normally be the limiting factor.

Ole Tange
  • 5,099