Two seemingly identical strings with newlines not equal

Question

I am trying to convert a list of quoted strings, separated by commas, into list of strings separated by newlines using bash and sed.

Here is an example of what I am doing:

#!/bin/bash

comma_to_newline() {
  sed -En $'s/[ \t]*"([^"]*)",?[ \t]*/\\1\\\n/gp'
}

input='"one","two","three"'
expected="one\ntwo\nthree" 
result="$( echo "${input}" | comma_to_newline )"

echo "Expected: <${expected}>"
echo "Result: <${result}>"

if [ "${result}" = "${expected}" ]; then
  echo "EQUAL!"
else
  echo "NOT EQUAL!"
fi

And the output I am getting is:

Expected: <one
two
three>
Result: <one
two
three>
NOT EQUAL!

I know it has something to do with the newlines characters, but I can't work out what. If I replace the newlines with some other string, such as XXX, it works fine and bash reports the strings as being equal.

That's really what you get for the Expected printout? Not `Expected: `? — Shawn, May 01 '20 at 00:03
Yup, that is what I get. I am running: `GNU bash, version 3.2.57(1)-release-(x86_64-apple-darwin18)` on Mac OS. But getting `` on `GNU bash, version 5.0.3(1)-release (i686-pc-linux-gnu)` on Debian — njh, May 01 '20 at 00:15
I was expecting the problem to be with my sed function, but it turns out it was a problem with my expected string and an old version bash! `expected=$'one\ntwo\nthree'` works fine! — njh, May 01 '20 at 00:19
Note that your variable `expected` does not contain any newline character. You can verify this by writing a `xxd <<<$expected` just after setting the variable. — user1934428, May 01 '20 at 07:04

score 1 · Accepted Answer · answered May 02 '20 at 00:27

Prompted by the comments on my question, I managed to work out what was going on. I was so focussed on coming up with a working sed expression and ensuring that result was correct, that I failed to noticed that the expected string was incorrect.

In order to use \n newlines in a bash string, you have to use the $'one\ntwo\nthree' syntax - see How can I have a newline in a string in sh? for other solutions.
I was developing against bash version 3.2.57 (the version that comes with Mac OS 10.14.6). When assigning a variable using expected="one\ntwo\nthree" then echoing it, they were being displayed as newlines in the console. Newer versions of bash display these strings as escaped - so I assume it is a bug that has been fixed in later versions of bash.

score 0 · Answer 2 · answered May 01 '20 at 06:15

0

For diagnosing seemingly identical strings, try combining side-by-side diff output with a one char per line hexdump format. Replace:

else
  echo "NOT EQUAL!"
fi

...with:

else
    echo "NOT EQUAL!"
    diff -y \
    <(hexdump -v  -e '/1  "%_ad#  "' -e '/1 " _%_u\_\n"' <<< "${expected}") \
    <(hexdump -v  -e '/1  "%_ad#  "' -e '/1 " _%_u\_\n"' <<< "${result}")
fi

answered May 01 '20 at 06:15

agc

7,973
2
29
50

This would be a good answer if there were weird invisible / unicode characters involved. But my problem turned out to be because what bash was printing to stdout was different to its internal representation. And I suspect an old bug in Bash 3.2.57. – njh May 01 '20 at 08:33
1

@njh, You seem to have nailed it down, so please consider [answering your own question](https://meta.stackexchange.com/q/17845/334345). Re _"...a good answer **if**..."_: either way, answers like this help test whether such an *"if"* is true or false. – agc May 01 '20 at 17:31

score 0 · Answer 3 · answered May 01 '20 at 13:45

There is extra new line character \n in string returing from your function.

Octal dump

$echo '"one","two","three"' | sed -En $'s/[ \t]*"([^"]*)",?[ \t]*/\\1\\\n/gp' | od -c 
0000000   o   n   e  \n   t   w   o  \n   t   h   r   e   e  \n  \n
0000017
$echo "one\ntwo\nthree"  |  od -c 
0000000   o   n   e   \   n   t   w   o   \   n   t   h   r   e   e  \n
0000020
$

Also, use echo -e

$echo "one\ntwo\nthree"  
one\ntwo\nthree
$echo -e "one\ntwo\nthree"  
one
two
three
$

From man page

-e enable interpretation of backslash escapes

Two seemingly identical strings with newlines not equal

3 Answers3