How to avoid bash command substitution to remove the newline character?

Question

To speed up some bash script execution, I would like to keep the result of a command in a variable using command substitution, but the command substitution replaces the 0x0A newline character by a space. For example:

a=`df -H`

or

a=$( df -H )

When I want to process further $a, the newline characters are replaced by a space and all the lines are now on one line, which is much harder to grep:

echo $a

What would be the easy tricks to avoid the newline character being removed by the command substitution?

Answer 1

Non-trailing newlines are not removed

The newlines you are looking for are there, you just don't see them, because you use echo without quoting the variable.

Validation:

$ a=$( df -H )
$ echo $a
Filesystem Size Used Avail Use% Mounted on /dev/sda3 276G 50G 213G 19% / udev 2.1G 4.1k 2.1G 1% /dev tmpfs 832M 820k 832M 1% /run none 5.3M 0 5.3M 0% /run/lock none 2.1G 320k 2.1G 1% /run/shm
$ echo "$a"
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       276G   50G  213G  19% /
udev            2.1G  4.1k  2.1G   1% /dev
tmpfs           832M  820k  832M   1% /run
none            5.3M     0  5.3M   0% /run/lock
none            2.1G  320k  2.1G   1% /run/shm
$ 

Trailing newlines are removed

As @user4815162342 correctly pointed out, although newlines within the output are not removed, trailing newlines are removed with command substitution. See experiment below:

$ a=$'test\n\n'
$ echo "$a"
test


$ b=$(echo "$a")
$ echo "$b"
test
$

In most cases this does not matter, because echo will add the removed newline (unless it is invoked with the -n option), but there are some edge cases where there are more that one trailing newlines in the output of a program, and they are significant for some reason.

Workarounds

1. Add dummy character

In these case, as @Scrutinizer mentioned, you can use the following workaround:

$ a=$(printf 'test\n\n'; printf x); a=${a%x}
$ echo "$a"
test


$ 

Explanation: Character x is added to the output (using printf x), after the newlines. Since the newlines are not trailing any more, they are not removed by the command substitution. The next step is to remove the x we added, using the % operator in ${a%x}. Now we have the original output, with all newlines present!!!

2. Read using process substitution

Instead of using command substitution to assign the output of a program to variable, we can instead use process substitution to feed the output of the program to the read built-in command (credit to @ormaaj). Process substitution preserves all newlines. Reading the output to a variable is a bit tricky, but you can do it like this:

$ IFS= read -rd '' var < <( printf 'test\n\n' ) 
$ echo "$var"
test


$ 

Explanation:

• We set the internal field separator for the read command to null, with IFS=. Otherwise read would not assign the entire output to var, but only the first token.
• We invoke read with options -rd ''. The r is for preventing the backslash to act as a special character, and with d '' set the delimiter to nothing, so that read reads the entire output, instead of just the first line.

3. Read from a pipe

Instead of using command or process substitution to assign the output of a program to variable, we can instead pipe the output of the program to the read command (credit to @ormaaj). Piping also preserves all newlines. Note however, that this time we set the lastpipe shell optional behavior, using the shopt builtin. This is required, so that the read command is executed in the current shell environment. Otherwise, the variable will be assigned in a subshell, and it will not be accessible from the rest of the script.

$ cat test.sh 
#!/bin/bash
shopt -s lastpipe
printf "test\n\n" | IFS= read -rd '' var
echo "$var"
$ ./test.sh 
test


$

Answer 2

I was trying to wrap my head around this because I was using bash to stream in the result of running the interpreter on an F# script. After some trial and error, this turned out to solve the problem:

$ cat fsi.ch
#!/bin/bash
echo "$(fsharpi --quiet --exec --nologo $1)"

$ fsi.ch messages.fsx
Welcome to my program. Choose from the menu:
new | show | remove

Assuming, of course that you need to run a terminal program. Hope this helps.

Answer 3

Another "neat trick" is to use the carriage return character, which prevents the newline from being stripped but doesn't add anything to the output:

$ my_func_1 () {
>     echo "This newline is squashed"
> }
$ my_func_2 () {
>     echo "This newline is not squashed"
>     echo -n $'\r'
> }
$ echo -n "$(my_func_1)" && echo -n "$(my_func_2)" && echo done
This newline is squashedThis newline is not squashed
done
$

But buyer beware: as mentioned in the comments this can work nicely for output that is simply going to the terminal, but if you are passing this on to another process you might confuse it as it probably won't be expecting the weird terminating '\r'.