Shell redirection i/o order

Question

I'm playing with i/o shell redirection. The commands I've tried (in bash):

ls -al *.xyz 2>&1 1> files.lst

and

ls -al *.xyz 1> files.lst 2>&1

There is no any *.xyz file in current folder.

These commands gives me the different results. The first command shows an error message ls: *.xyz: No such file or directory on the screen. But the second one prints this error message to the file. Why did the first command failed to write an err output to the file?

Answer 1

The Bash manual has a clear example (similar to yours) to show that the order matters and also explains the difference. Here's the relevant part excerpted (emphasis mine):

Note that the order of redirections is significant. For example, the command

ls > dirlist 2>&1

directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command

ls 2>&1 > dirlist

directs only the standard output to file dirlist, because the standard error was made a copy of the standard output before the standard output was redirected to dirlist.

This post explains it from the POSIX viewpoint.

Confusions happen due to a key difference. > redirects not by making left operand (stderr) point to right operand (stdout) but by making a copy of the right operand and assigning it to the left. Conceptually, assignment by copy and not by reference.

So reading from left-to-right which is how this is interpreted by Bash: ls > dirlist 2>&1 means redirect stdout to the file dirlist, followed by redirection of stderr to whatever stdout is currently (which is already the file dirlist). However, ls 2>&1 > dirlist would redirect stderr to whatever stdout is currently (which is the screen/terminal) and then redirect stdout to dirlist.

Answer 2

Redirections are:

• processed from left to right.
• interpreted iteratively:
  • an earlier redirection can affect a later one:
    • if an earlier redirection has redirected a given stream (identified by a file descriptor number, such as 1 for stdout (the default), and 2 for stderr), later redirections targeting that stream refer to the already-redirected version.
  • but not vice versa - a later redirection has no retroactive effect on the the target of an earlier redirection:
    • e.g., if you specify file descriptor 1 as the target in an earlier redirection, what 1 means at that time is locked in, even if 1 is redirected later.
• Note, however, that output isn't actually sent until all redirections are in place, and that any redirection-target output files are created or truncated before command execution begins (this is the reason why you can't read from and redirect output to the same file with a single command).

Applied to the example from the question:

• >file 2>&1:

  • >file first redirects stdout (file descriptor 1, implied by not prefixing > with a file descriptor number) to output file file
  • 2>&1 then redirects stderr (2) to the already redirected stdout (1).
  • The net effect is that both original streams end up in file.

• 2>&1 >file:

  • 2>&1 first redirects stderr to the then-original stdout; since file descriptor 2 participates in no further redirections, stderr output will therefore go to whatever stdout was defined as at that point - i.e., the original stdout, given that this is the first redirection.
    • Technically, the original stdout file descriptor is duplicated, and that duplicate is what stderr then refers to, which explains why it isn't affected by a later redirection of stdout.
  • >file then redirects the original stdout to file - but that has no effect anymore on the already locked-in redirection of stderr.
  • The net effect is that only sent-directly-to-stdout output is captured in file, while sent-to-stderr output is output to (the original, unredirected) stdout.

Answer 3

This error:

ls: *.xyz: No such file or directory

is being written on stderr by ls binary.

However in this command:

ls -al *.xyz 2>&1 1> files.lst

You're first redirecting stderr to stdout which by default goes to tty (terminal)

And then you're redirecting stdout to a file files.lst, however remember that stderr doesn't redirected to file since you have stderr to stdout redirection before stdout to file redirection. Your stderr still gets written to tty in this case.

However in 2nd case you change the order of redirections (first stdout to file and then stderr to stdout) and that rightly redirects stderr to a file which is also being used by stdout.

Answer 4

Because order does matter. In the first case, you first redirect stderr (2) to stdout (1). Then you redirect (1) to a file. But stderr (2) is still pointed to stdout of the shell running the command. Pointing (1) to a file in this case doesn't change the output device that (2) is directed at, so it still goes to terminal.

In the second case, you redirect stdout (1) to a file. Then you point stderr (2) to the same place 1 is pointed, which is the file, so the error message goes to the file.

Answer 5

Redirections are processed from left to right

they are the same:

• my_cmd 1>a_file
• my_cmd >a_file

---

My method to remember the whole thing

let's imagine we are playing a game. There is a bridge with 2 broken parts.

1. If we first place a block named 2>&1 to repair the first broken part, the ball named stderrr can arrive the place named stdout, but since the second part is still broken, stderrr will fall down to a river named tty (to your screen) .Then we place a block named 1>a_file to repair the second broken part, the ball named stdout can arrive the place named a_file

2. If we first place a block named 1>a_file to repair the second broken part and then repair the first broken part with 2>&1, the ball stderrr will not fall down to the river tty

Answer 6

First

$ ls *.xyz 2>&1 >files.lst
ls: cannot access '*.xyz': No such file or directory
$ cat files.lst

This writes the ls error message to the terminal because when 2>&1 copies stdout to stderr they both still points to the terminal. files.lst is empty because stderr is not pointing to this file but to the terminal.

Next

$ ls *.xyz >files.lst 2>&1
$ cat files.lst
ls: cannot access '*.xyz': No such file or directory

Here >files.lst redirects stdout to the file files.lst. Then 2>&1 redirects stderr to stdout which now points to files.lst.

So when you cat the file it now contains the ls error message.

Answer 7

ls -al *.xyz 2>&1 1> files.lst

In the first example, 2>&1 redirection just points stderr to screen.

ls -al *.xyz 1> files.lst 2>&1

Here though, this has an effect of ls -al *.xyz &>files.lst. Both stdout and stderr are passed to files.lst file.

The difference is the order of redirections.