1

I'm using simple function to make my script more verbose.

cmd() {
  echo "[#] $*" >&2
  "$@"
}

When calling script with option -v all commands goes through cmd function:

build() {
  cmd yarn webpack --config webpack.prod.js
}

copy_to_server() {
  cmd tar -cf - -C ./dist . | ssh root@mydomain.com tar -xf - -C /path/to/desirable/dir
}


main() {
  build
  copy_to_server
}


main

Everything works nice except command with pipe. It even echoed only up to dot before pipe sign. Is there any chance to make it properly work?

yar83
  • 13

1 Answers1

1

Explanation

In this piece of code:

cmd tar -cf - -C ./dist . | ssh root@mydomain.com tar -xf - -C /path/to/desirable/dir

cmd does not see the pipe symbol and beyond. If cmd wasn't there, tar wouldn't see the pipe symbol either.

Out of the box solutions

Do not reinvent the wheel. A shell (not only Bash but any POSIX-like shell) gives you at least two options that resemble what your cmd does:

-v
The shell shall write its input to standard error as it is read.

-x The shell shall write to standard error a trace for each command after it expands the command and before it executes it. […]

(source)

You enable the options with set -v and set -x; you disable with set +v and set +x respectively. You can enable or disable both at once with set -vx or set +vx.

In case of set -v, your pipeline will be printed as-is. set -x will print tar and ssh separately, but it will expand it first; on the other hand iw won't show you redirections. The below script shows these differences (there may be more of them):

#!/bin/sh
set -vx
foo=bar
echo "$foo" | wc -c 2>/dev/null

The output will be (# comments mine):

foo=bar                          # from set -v,   to stderr
+ foo=bar                        # from set -x,   to stderr
echo "$foo" | wc -c 2>/dev/null  # from set -v,   to stderr
+ echo bar                       # from set -x,   to stderr
+ wc -c                          # from set -x,   to stderr
4                                # actual output, to stdout

Also note set -v prints shell code as it is read, this results in at least two quirks:

  1. Some snippets have to be read at once, e.g. while … done or if … fi (because there may be redirection(s) after done or fi), so set -v will show each such snippet as a whole, once.

  2. A command like sh -c 'set -v; echo foo' will "ignore" set -v because at the time set -v gets executed all the input has already been read. This will also happen if you use a newline instead of ;; or with sh -vc 'echo foo'. In a script set -v should work well.

An advantage is you can (possibly conditionally) run set -v or set -x (or set -vx) early in your script and there is no need to modify the rest of the code. A disadvantage is the whole later code (until set +v or so) will be affected, while with your cmd (if it worked exactly as you wish) you can apply cmd to specific lines only.

Custom solution

I can imagine neither set -v nor set -x (nor set -vx) is exactly what you want. There is a way to make your cmd print pipelines like tar … | ssh …. Take this function:

cmd() {
   [ "$debug" = 1 ] && (IFS=' '; printf '%s\n' "$*" >&2)
   eval "$@"
}

and use it like this:

debug=1
cmd 'tar -cf - -C ./dist . | ssh root@mydomain.com tar -xf - -C /path/to/desirable/dir'

With debug=1 and debug=something_else you can turn printing from cmd on or off.

But you need to be very careful. I deliberately built cmd so it accepts multiple arguments. The support for multiple arguments allows you to simply run:

cmd echo foo

or

cmd echo foo \| wc -c

where only | needs to be escaped (or quoted). You need to be careful with expansions and quoting though (even in simple cases). The code is parsed before cmd runs and again when eval runs. So this:

foo='$(echo 123456789)'
cmd echo "$foo"

will expand $foo before cmd and then eval will expand $(echo 123456789) and the output will be 123456789. Note it could be foo='$(reboot)', while by cmd echo "$foo" you clearly don't want to reboot.

And while echo ">very-important-file" is harmless, cmd echo ">very-important-file" will truncate very-important-file to zero size. The quotes will be removed before cmd runs and they won't appear in what eval evaluates, so the command will become echo >very-important-file.

A firm way to avoid such mishaps is to use cmd with exactly one single-quoted argument. This is especially important if some part (like the foo variable) is not under your total control (e.g. it comes from the environment or from read foo). A safe usage looks like this:

cmd 'echo "$foo"'

or

cmd 'echo ">very-important-file"'

Passing everything as a single-quoted argument complicates quoting in cases when you actually need single-quotes before introducing cmd, but since you should prevent expansion at this stage, some additional quoting and/or escaping is inevitable; or else you risk executing the wrong code.

A general procedure to single-quote any string in a shell (even if the string contains single-quotes) is:

  • replace every ' with '"'"' or with '\'',
  • embrace the whole resulting string with single-quotes.

Finally you add cmd in front. The procedure is totally "mechanical", there are no quirks, no exceptions, it requires no further insight, so it can be easily automated (and Bash itself can do this, but note the linked question is for the command line of interactive Bash and it won't help you when writing scripts in a text editor; a powerful enough editor should allow you to automate this in its own way though).

Now you can pass any snippet to cmd, no matter if it contains |, &&, [[, whole if clause, newlines, redirections or whatever. Examples:

cmd 'echo foo && echo bar; echo "$PATH" | dd conv=ucase 2>/dev/null'

cmd '
echo '\''$PATH'\'' is
echo "$PATH"
'

Inside cmd we use $* and $@. It's true $* uses the first character of $IFS, therefore we fix IFS where we use $* with printf; but using $* with eval while similarly fixing IFS would be wrong because it would affect IFS for the shell code we want to execute; fortunately eval works fine with "$@". Our cmd should work with any IFS. Example:

IFS=.
cmd read a b c

Provide 1.2.3Enter and then check (with echo "$a") if $a is 1. (Note if you do this in an interactive shell then the modified IFS will stay.)

IFS=. cmd read a b c and cmd IFS=. read a b c should also work as expected. In neither case the modified IFS will break our cmd. Note in these examples I deliberately passed multiple arguments to cmd to truly test if $*, $@ and the whole function behave well.

Summary

Try set -v and/or set -x, maybe they will be enough for you. If not, use our custom cmd, but then you really need to quote and/or escape right.

Kamil Maciorowski
  • 81,893