23

I have a bash script that contains the following two commands:

ssh host tail -f /some/file | awk ..... > /some/file &

ssh host tail -f /some/file | grep .... > /some/file &

How can I make the output of both commands be directed into the same file?

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
user2864207
  • 333
  • 1
  • 6
  • 12
  • 2
    Keep in mind that if both commands are running in the background, you cannot predict how the output from the two commands will be interleaved in `/some/file`. – chepner Dec 03 '13 at 15:48

3 Answers3

42

Either use 'append' with >> or use braces to encompass the I/O redirections, or (occasionally) use exec:

ssh host tail -f /some/file | awk ..... >  /some/file &
ssh host tail -f /some/file | grep .... >> /some/file &

or:

{
ssh host tail -f /some/file | awk ..... &
ssh host tail -f /some/file | grep .... &
} > /some/file

or:

exec > /some/file
ssh host tail -f /some/file | awk ..... &
ssh host tail -f /some/file | grep .... &

After the exec, the standard output of the script as a whole goes to /some/file. I seldom use this technique; I usually use the { ...; } technique instead.

Note: You do have to be careful with the braces notation. What I showed will work. Trying to flatten it onto one line requires you to treat the { as if it were a command (followed by a space, for example) and also to treat the } as if it were a command. You must have a command terminator before the } — I used a newline, but an & for background or ; would work too.

Thus:

{ command1;  command2;  } >/some/file
{ command1 & command2 & } >/some/file

I also have not addressed the issue of why you have two separate tail -f operations running on a single remote file and why you are not using awk power as a super-grep to handle it all in one — I've only addressed the surface question of how to redirect the I/O of the two commands to one file.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
  • none of this works. I also tried { command1 ; command 2 } > /some/file, but it doesn't work – user2864207 Dec 03 '13 at 16:10
  • 3
    @user2864207, note that the one-line use of `{ list of commands; inside braces; }` **must** end with a semicolon, as illustrated in Jonathan's last sentence. – glenn jackman Dec 03 '13 at 16:14
  • You do have to be careful with the braces notation. What I showed will work. Trying to flatten it onto one line (why — why on earth would two separate pipelines be flattened onto one line? have you no respect for readability or comprehensibility?) requires you to treat the `{` as if it were a command (followed by a space, for example) and also to treat the `}` as if it were a command. You must have a command terminator before the `}` — I used a newline, but an `&` for background or `;` would work too. – Jonathan Leffler Dec 03 '13 at 16:22
  • @JonathanLeffler Could you point to where the problem you described above with flattening onto one line would be explained in more detail? – Piotr Dobrogost Feb 03 '17 at 15:40
  • 1
    @PiotrDobrogost: I'm not sure what extra detail you're looking for. Any standard shell text book should cover the issue — it was a facet of the Bourne shell in the early 80s, and was continued into the POSIX shell, and therefore is a part of Korn shell and Bash. You could study the POSIX specification of the [shell language](http://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html); you'd find it codified there (but it isn't an easy read). Which aspect of the examples in the question and comment aren't clear enough? – Jonathan Leffler Feb 03 '17 at 15:47
  • @JonathanLeffler I was curious about *(...) requires you to treat the { as if it were a command (...)* statement specifically as it's rather strange to treat something that should be part of syntax as a command. I was wondering if it's something similar to `[` which is a real binary? – Piotr Dobrogost Feb 03 '17 at 16:24
  • @PiotrDobrogost: it is loosely similar to `[`, but `{` is not a binary. In fact, it is a [reserved word](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_04) in the shell — which I'd forgotten or had not appreciated. See also the information on [compound commands](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04). – Jonathan Leffler Feb 03 '17 at 19:56
  • @PiotrDobrogost: The GNU Bash manual has a section on [Command Grouping](https://www.gnu.org/software/bash/manual/bash.html#Command-Grouping) which states more or less what I stated — it is corroboration, but not a lot more informative (though it is not clear what else could be used to explain it; there isn't much more to be said unless you grovel through the shell grammar). – Jonathan Leffler Dec 16 '19 at 23:13
  • 1
    As a noob, this taught me so much. Thanks for the help. – Ender Che Jun 04 '20 at 22:41
18

Note you can reduce the number of ssh calls:

{  ssh host tail -f /some/file | 
     tee >(awk ...) >(grep ...) >/dev/null
} > /some/file &

example:

{ echo foobar | tee >(sed 's/foo/FOO/') >(sed 's/bar/BAR/') > /dev/null; } > outputfile
cat outputfile 

fooBAR
FOObar
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
2

The best answer to this is to probably get rid of the ssh .... | grep ... line, and modify the awk script in the other command to add the functionality you were getting from the grep command...

That will get rid of any interleaving issues as a bonus side-effect.

twalberg
  • 59,951
  • 11
  • 89
  • 84