2

I'd like to pipe the output of 1 command into multiple pipes and pipe outputs from multiple commands into 1 single command.

An example of split:

# Single output to multiple pipes
echo "This is a sentence" | split | cut -d ' ' -f 1,3 > file1.txt
                                  | cut -d ' ' -f 2,4 > file2.txt

In the above example of split, the output "This is a sentence" is split piped into 2 cuts and 2 different files.

An example of join:

Hello(){
   echo "Hello $1 and $2"
}

echo "Alice" | 
echo "Bob"   | join | Hello
# output: "Hello Alice and Bob"

In the above example of join takes the outputs from 2 different pipes and give them to the function Hello as 2 piped inputs.

An example of split and join used together:

# split the input into multiple pipes
echo "Alice Bob Charlie Dave" | split

# 2 separate split pipes are processed and then joined into a single pipe
split | cut -d ' ' -f 1,3 | join
split | cut -d ' ' -f 2,4 | join

Hello(){
   echo "$1 loves $2"
}

join | Hello

# Output:
# Alice Charlie loves Bob Dave

The above example implements both split and join to have multiple pipes processed in parallel. split is "single input, multiple outputs", join is "multiple inputs, single output".

What's the best way to implement the demonstrated split and join in a shell/bash script?

ChocolateOverflow
  • 269

2 Answers2

1

You may use GNU Parallel for split:

echo "This is a sentence" |
  parallel --tee --pipe -q cut -d ' ' -f {} ::: 1,3 2,4

echo "This is a sentence" |
  parallel --tee --pipe "cut -d ' ' -f {} >file{#}" ::: 1,3 2,4
Ole Tange
  • 5,099
1

Another approach is to use tee, or in this case likely better use awk, perl etc.

Or to be wonky ...

echo "Jane Bob Alice Joe" |
(tee >(
  cut -d ' ' -f1,3 -z >&2 && printf ' loves ' >&2) |
  cut -d ' ' -f2,4
) 2>&1 | 
tr -d '\0'

Jane Alice loves Bob Joe

Split

Split using tee + process substitution / subshell.

echo "This is a sentence" | 
tee >(cut -d ' ' -f1,3 >file1.txt) | 
cut -d ' ' -f2,4 >file2.txt

...


echo "This is a sentence with some more words" |
tee 

    >(cut -d ' ' -f1,3 >file1.txt) 

    >(cut -d ' ' -f2,4 >file2.txt) 

    >(cut -d ' ' -f3,5 >file3.txt) 

    >(cut -d ...

But with for example awk we can do more logic whilst also keeping the processes and parsing down.

awk '{ 
    print $1,$3 > "file1.txt"
    print $2,$4 > "file2.txt"
}'

... arbitrary length of input:


awk '{
    for (i = 1; i < NF - 1; ++i)
        print $i, $(i + 2)  > sprintf("file%02d.txt", i)
}'

Join

As for join it is a bit unclear what the goal really is. Some variants:

Starting out with:

#! /bin/bash -
awk '{
    for (i = 1; i < NF - 1; ++i)
        print $i, $(i + 2)  > sprintf("file%02d.txt", i)
}'

... or simply

#! /bin/awk -f
{
    for (i = 1; i < NF - 1; ++i)
        print $i, $(i + 2)  > sprintf("file%02d.txt", i)
}

$ ./splitinput <<<"Alice Jim Jane Carl An Fred Adler Morgan"
$ tail file0*
==> file01.txt <==
Alice Jane

==> file02.txt <==
Jim Carl


==> file03.txt <==
Jane An


==> file04.txt <==
Carl Fred


==> file05.txt <==
An Adler


==> file06.txt <==
Fred Morgan
  • Using paste:
$ paste -d ' ' file01.txt <(echo loves) file02.txt
Alice Jane loves Jim Carl
  • Using bash read single shot:
hello() {
    printf '%s loves %s\n' "$1" "$2"
}
IFS= read -r name1<"$1"
IFS= read -r name2<"$2"
hello "$name1" "$name2"

./script file01.txt file02.txt

Multi set operation

  • Using bash read in combination with cat:
names=()
while IFS= read -r name; do
    names+=("$name")
done< <(cat $@)

Which can be simplified as:


IFS=$'\n' read -rd '' -a names< <(cat file*)

Gives an array of names to work with:

printf '# Name: %s\n' "${names[@]}"
# Name: Alice Jane
# Name: Jim Carl
# Name: Jane An
# Name: Carl Fred
# Name: An Adler
# Name: Fred Morgan

printf ';; bash for\n'
for ((i = 0; i < ${#names[@]}; i += 2)) do
    hello "${names[@]:i:i+2}"
done

Using a loop to process all files with read:

printf ';; bash while argv\n'
while [ $# -gt 0 ]; do
    IFS= read -r name1<"$1"
    shift
    IFS= read -r name2<"$1"
    shift
    hello "$name1" "$name2"
done

Using awk to process all files:

printf ';; awk\n'
awk -v adv=loves '
{
    if (NR % 2)
        name=$0
    else
        print name,adv,$0
}' "$@"

Alice Jane loves Jim Carl
Jane An loves Carl Fred
An Adler loves Fred Morgan

Or, to get fancy.

hello2() {
    fmt -uw32 <<-EOH
    $1 loves $2, $2 loves $3 and $3 loves $2.

$5 is in love with $4 and $6 is all alone,
but has a secret crush on both $1 and $3.
EOH



}
hello2 "${names[@]}"

Alice Jane loves Jim Carl, Jim
Carl loves Jane An and Jane An
loves Jim Carl.

An Adler is in love with Carl
Fred and Fred Morgan is all
alone, but has a secret crush
on both Alice Jane and Jane An.
user3342816
  • 283