3

Inside a Makefile I run a shell command which I want to pass a NULL byte as argument. The following attempt fails:

echo $(shell /bin/echo -n $$'\x00' | ruby -e "puts STDIN.read.inspect")

It generates:

echo "$\\x00"

Instead I expected:

echo "\u0000"

How do I properly escape such a NULL byte?

johannes
  • 7,262
  • 5
  • 38
  • 57

5 Answers5

11

echo disables interpretation of backslash escapes by default. You need to supply the -e option to enable it.

$ echo -ne "\x00" | ruby -e "puts STDIN.read.inspect"
"\u0000"
devnull
  • 118,548
  • 33
  • 236
  • 227
  • I am not interested in passing null bytes over pipes, but in passing them directly inside arguments. – johannes Jun 14 '13 at 12:56
  • 1
    Oh! So you need to state clearly what you intend to do. The various answers attempt to address the issue you **describe**. – devnull Jun 14 '13 at 13:01
  • Let me quote from my question: >which I want to pass a NULL byte as **argument** – johannes Jun 14 '13 at 13:10
  • Which was substantiated with an example. – devnull Jun 14 '13 at 13:13
  • Feel free to delete your answer if you dislike having written a downvoted answer: http://meta.stackexchange.com/questions/91124/etiquette-on-removing-wrong-answer-after-a-downvote – johannes Jun 14 '13 at 13:19
  • `bash` (and I believe most `sh`?) ignores null bytes in command substitution, so it's a no-go unless you pipe. There's no need for `echo`. `printf '\0'` is more reliable – CervEd Dec 18 '21 at 23:17
8

Due to the execve(2) semantics it is not possible to pass a string containing a null byte as argument. Each argument string is terminated by null byte, therefore making it impossible to distinguish between the contained null byte and the end of the string.

johannes
  • 7,262
  • 5
  • 38
  • 57
2

These uses of echo are totally non-portable. Use printf, it's much easier to use for anything other than the simplest strings, and much more portable.

$ cat makefile
all:
        printf '\0' > foo.out
        od -a foo.out

$ make
printf '\0' > foo.out
od -a foo.out
0000000 nul
0000001
MadScientist
  • 92,819
  • 9
  • 109
  • 136
  • If you want a good answer, ask a good question. Your example showed echoing a NUL byte. I showed you how to generate a NUL byte. If that's not what you wanted, then ask the question you want answered. If you really wanted to know how to split the output of `find -print0` into individual arguments in the shell, then ask THAT question. Then we could have all immediately explained that such a thing is not useful in the context of breaking up words in a shell script. – MadScientist Jun 14 '13 at 13:37
  • 1
    It seems I am unable to come up with a minimal example not drawing the readers thoughts away from my intention. You have enough reputation to edit my question. Please show me an example matching exactly my questions intention of passing a NULL byte as an argument. – johannes Jun 15 '13 at 15:55
  • 1
    Not being able to come up with a good example is the first hint of trouble. At least describe at a higher level what you're trying to accomplish. Elsewhere you mention sending the output of `find -print0` into `cut`... but then what? Where does the output of `cut` go? Wherever it goes, obviously it won't be delimited by NUL bytes anymore so essentially `-print0` is useless. Fundamentally, `-print0` is only helpful if the final consumer of the output can _itself_ split on NUL bytes: the shell cannot do that. Here we don't know what that consumer is so we can't advise. – MadScientist Jun 15 '13 at 18:50
  • 1
    I assumed it was a good thing to limit the question to the small specific detail I have trouble with. Maybe next time I will ask: "I am trying to achieve world peace. The 1.000.000 lines of code I have written can be viewed at http://bla.bla.bla/. Please fix my code!!!" – johannes Jun 16 '13 at 18:18
1

You can't use NUL as argument in bash

You can't use $'\0' as an argument, store it as a variable or using command substitution $(printf '\0') since bash (and most shells?) use C-strings that are null terminated. The leading string before NUL is interpreted as the string and the trailing string discarded.

You can only input using pipes - printf '\0' | cat -v or letting the resulting program use a file for input.

Use another means of input

Most programs that work on input with line strings NUL strings (xargs, cut, ...) typically have a -z flag. This is primarily used when dealing with paths as a character may contain ANY character EXCEPT NUL.

Programs like find and git ls-files support outputting this format, usually in the form of a -print0 or -0 flag.

Programs like sed, tr, bash et. al. use special escape characters like \0, \x0, \x00 to generate NUL bytes.

Massage the input

OP originally seems to have wanted to know how to use cut with a NUL delimiter. The problem is typically that something is separated using \n, where \n is a valid part of the values and not a line-separator (typically in paths).

Say you have a situation where you group files, each separated by a NUL character, and the groups separated by \n.

# Temporary mock output with NUL columns and newline rows
printf 'foo\0bar\nbar\0\nFOO\0BAR\0\n' > /tmp/$$.output

A work-around is to get creative with a combination of sed, awk or tr to massage the output to something that suits our input/commands.

our.sed

#!/usr/bin/sed -nf

# beginning
:x 
# match \0\n
/\x0$/ {
 # Change \0\n to \n
 s|\x0$|\n|g
 # print
 p
 # delete
 d
}
# match \n with no leading \0
/[^\x0]$/ {
 # change \0 to \1
 s|\x0|\x1|g
 # read next line
 N
 # branch to beginning
 bx
}

In this scenario we map:

  • \0\n => \n
  • \0 not followed by \n => \1
    While a valid character in a filename, it's unlikely to pose a problem.
# Change NUL to another unlikely to be used control character
sed -f our.sed /tmp/$$.output |\
  cut -d $'\x1' -f 2

output

bar
BAR
CervEd
  • 3,306
  • 28
  • 25
0

If anyone else came here looking how to escape a null via a shell command in ruby backticks:

irb(main):024:0> `curl --silent http://some-website-or-stream.com | sed 's/\\x0//g' 1>&2`
=> ""
Blaskovicz
  • 6,122
  • 7
  • 41
  • 50