No error message from curl

Question

I would like to know why the following bash script swallows the error message.

#!/bin/sh

set -eu

LATEST=$(curl -s https://api.github.com/repos/dnote-io/cli/tags | grep -Eo '"name": "v\d*\.\d*\.\d*",'  | head -n 1 | sed 's/[," ]//g' | cut -d ':' -f 2)

if [ -z $LATEST ]; then
    echo "Error fetching latest version. Please try again."
    exit 1
fi

....

Basically, the script fetches something and assigns the result to a variable LATEST. But when the fetching goes wrong, the script does not print the actual error message from curl.

Any ideas or suggestions on how to not ignore the error?

Answer 1

What error message?

-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.

Use -sS instead.

-S, --show-error
When used with -s it makes curl show an error message if it fails.

There are other problems in your script:

• If curl fails, the script continues without noticing. This is because the failure of a command on the left-hand side of a pipeline is ignored. Usually a failure will lead to an empty output in LATEST, but this may not always be the case, if curl manages to connect but the connection is interrupted while it's downloading the data.
  In bash, but not in sh, you can capture the failure of the left-hand side of a pipe by setting the pipefail option. If you want to keep going, check PIPESTATUS, but note that you'd have to do this inside the command substitution.
• [ -z $LATEST ] is wrong because the value of LATEST is split into whitespace-delimited words which are interpreted as wildcard patterns. It'll work if the value doesn't contain any special characters, but this is fragile, especially since you're parsing downloaded content. See Why does my shell script choke on whitespace or other special characters? . Always use double quotes around variable and command substitutions, unless you know why you need to leave them off. Alternatively, in bash but not in sh, use the [[ … ]] syntax, which doesn't require double quotes (except on the right-hand side of comparison operators, and if you don't like remembering these exceptions, just use double quotes all the time).

#!/bin/bash
set -eu
set -o pipefail

LATEST=$(curl -s https://api.github.com/repos/dnote-io/cli/tags | grep -Eo '"name": "v\d*\.\d*\.\d*",'  | head -n 1 | sed 's/[," ]//g' | cut -d ':' -f 2)

if [ -z "$LATEST" ]; then
    echo >&2 "Download from github successful, but extracting the version failed."
    exit 1
fi

Answer 2

There are two factors here which both contribute.

1. You are running curl in a pipeline. The set -e functionality only applies to the last command in a pipe; this is how the shell is designed.

2. There is no error message from curl because you are running it with -s.

Capturing the output from a command and checking its exit status at the same time is easy, but then you end up with the results in a shell variable. Depending on your scenario, this may or may not be cumbersome.

if curlout=$(curl -s https://api.github.com/repos/dnote-io/cli/tags); then
    # curl succeeded; *now* parse the result
    latest="$(sed -n 's/.*"name": "\(v\[0-9]*\.\[0-9]*\.\[0-9]*\)",.*/!d;s//\1/;q' <<<"$curlout")"
else
    rc=$?
    echo "$0: curl failed: $rc" >&2
    exit "$rc"
fi

I refactored your long pipeline into a single sed script, since that was relatively easy to do. I still doubt you can find many grep -E implementations which actually understand the Perl regex-ism \dso maybe your code simply wasn't actually working. If the result is JSON, the proper solution would be to use a proper JSON parser like jq instead.

Also notice how your variables should be lower case; uppercase variables are generally reserved for system use.

Your shebang says #!/bin/sh so you can't use Bash features like pipefail with that. Perhaps you actually want #!/bin/bash in the shebang so you can use Bash features in your script (seeing as you tagged this question bash anyway).