233

It seems that newer versions of bash have the &> operator, which (if I understand correctly), redirects both stdout and stderr to a file (&>> appends to the file instead, as Adrian clarified).

What's the simplest way to achieve the same thing, but instead piping to another command?

For example, in this line:

cmd-doesnt-respect-difference-between-stdout-and-stderr | grep -i SomeError

I'd like the grep to match on content both in stdout and stderr (effectively, have them combined into one stream).

Note: this question is asking about piping, not redirecting - so it is not a duplicate of the question it's currently marked as a duplicate of.

Andrew Ferrier
  • 16,664
  • 13
  • 47
  • 76
  • See the second answer (https://stackoverflow.com/a/637834/1129642) on the linked question for the correct way to pipe both stdout and stderr. No need for another question. – Marki555 Jul 15 '16 at 08:25
  • 7
    @triplee Not an exact duplicate, is it? Pipe vs. redirect to file? – Benjamin W. Jul 26 '16 at 14:02
  • @BenjaminW There is at least one answer there which solves both scenarios, though it's not the accepted answer. This is a fairly common question so we could probably find a better duplicate, or ask a moderator to merge these - or even, in the worst case, craft an entirely new canonical for this topic. If you find a better dupe, by all means propose it. Thanks in advance. – tripleee Jul 26 '16 at 16:33
  • 16
    @tripleee Solves, yes, but none of the answers use the `|&` shortcut, which I think is by far the most convenient solution to "redirect both stdout and stderr to a pipe". – Benjamin W. Jul 26 '16 at 16:43
  • 3
    This is not a duplicate of the linked question, and it wasn't clear that Marko's answer did what I wanted. Also, it doesn't mention |&. Voting to reopen. – Martin Bonner supports Monica May 05 '17 at 13:23

3 Answers3

229

(Note that &>>file appends to a file while &> would redirect and overwrite a previously existing file.)

To combine stdout and stderr you would redirect the latter to the former using 1>&2. This redirects stdout (file descriptor 1) to stderr (file descriptor 2), e.g.:

$ { echo "stdout"; echo "stderr" 1>&2; } | grep -v std
stderr
$

stdout goes to stdout, stderr goes to stderr. grep only sees stdout, hence stderr prints to the terminal.

On the other hand:

$ { echo "stdout"; echo "stderr" 1>&2; } 2>&1 | grep -v std
$

After writing to both stdout and stderr, 2>&1 redirects stderr back to stdout and grep sees both strings on stdin, thus filters out both.

You can read more about redirection here.

Regarding your example (POSIX):

cmd-doesnt-respect-difference-between-stdout-and-stderr 2>&1 | grep -i SomeError

or, using >=bash-4:

cmd-doesnt-respect-difference-between-stdout-and-stderr |& grep -i SomeError
Eric Bolinger
  • 2,722
  • 1
  • 13
  • 22
Adrian Frühwirth
  • 42,970
  • 10
  • 60
  • 71
  • Thanks for the clarification on `&>>`. I've corrected my question. – Andrew Ferrier May 11 '13 at 13:16
  • 23
    I added your example to my answer, just in case it was not obvious based on my given examples. As a side-note, you can also use the bash-specific `|&` instead of `2>&1 |`. – Adrian Frühwirth May 11 '13 at 13:21
  • 17
    Side note about the shortcut `|&` proposed by @AdrianFrühwirth for future readers: this feature is only supported with `bash` version 4+. If you're using 3 or below, you have to stick with `2>&1 |`. – tomocafe Apr 21 '14 at 18:30
  • 6
    Bash redirection is [very well explained here](http://www.catonmat.net/blog/bash-one-liners-explained-part-three/). @AdrianFrühwirth has done a good job, the link pasted goes even further. Sometimes, I wish the official Bash documentation was that good. – David Andreoletti Mar 13 '15 at 03:07
  • 1
    @AdrianFrühwirth FYI: your first example references `2>&1` in the description but actually uses `1&2` in the code. – JESii Mar 17 '22 at 22:36
  • Good answer @AdrianFrühwirth, thanks! I wonder if you can comment on my new question https://stackoverflow.com/questions/73294378/curious-where-pytest-output-is-going-in-bash-command-with-entr which is related. Using your suggestions here I still can't find a solution. – Thomson Comer Aug 09 '22 at 15:34
189

Bash has a shorthand for 2>&1 |, namely |&, which pipes both stdout and stderr (see the manual):

cmd-doesnt-respect-difference-between-stdout-and-stderr |& grep -i SomeError

This was introduced in Bash 4.0, see the release notes.

Benjamin W.
  • 46,058
  • 19
  • 106
  • 116
  • 1
    Thanks for adding this for completeness. I'm going to keep the other answer correct as many folks are still using bash pre-4.0. But this is useful. – Andrew Ferrier May 09 '16 at 09:30
  • 15
    Most notably perhaps, the Bash that ships on macOS is too old to support this. – Flimm Jan 20 '17 at 10:21
  • 1
    @Flimm but the zsh isn't – Trenton Feb 13 '17 at 21:41
  • 1
    Since ksh uses |& for coproc, this seems like a bad choice for an unnecessary shorthand. I hate to see lines with a stack of dups and redirections as much as the next guy, but there's something to be said for being explicit....and I apologize that this comment doesn't add much. I just wanted to express distaste for the shorthand without downvoting an actually helpful answer, because it's good that people see this. I didn't know this, so thanks for making me aware. – Paul Hodges Oct 02 '17 at 14:59
  • 1
    @PaulHodges I agree that it's not portable - I mainly like using it for interactive Bash sessions to avoid typing too much. – Benjamin W. Oct 02 '17 at 16:40
  • Just a small note. That does not work for input from the keyboard, since that doesn't go out of either stdout nor stderr (apparently). – Scott Franco Apr 08 '23 at 18:48
  • @ScottFranco Do you have an example? Output has to go *somewhere*, but it might be buffered or similar. – Benjamin W. Apr 08 '23 at 18:53
  • Sure: samiam@samiam-h-pc-2:~/projects/pascal/pascal-p6/pascaline_tests$ p6 debug_test1 debug_test --debug --debugsrc &> junk hi there q The "hi there" and "q" were typed by me. Linux normally buffers the command line input (you can override this), and that does not go to stdout nor stderr. Apologies for messy example, it is the same with any program that takes input. – Scott Franco Apr 08 '23 at 19:49
1

If you want to pipe STDOUT and STDERR to different subshells instead of both to the same one (as |& / 2>&1 does), I use this:

producer \
  >  >( stdout_pipeline... ) \
  2> >( stderr_pipeline... ) \
  ...

While this uses the redirection operator, it uses pipes under the hood to connect the producer to the subshells.

You can process multiple output streams (2 is not the limit) this way, and also pipe (potentially multiple) input streams from subshells in a similar way with the penguin operator ([n]< <(...)).

Mark K Cowan
  • 1,755
  • 1
  • 20
  • 28