Using both stdout and stderr of curl in a subsequent jq command

Question

I want to do the following,

         command2(stdout)
        /                \
command1                  command4
        \                /
         command3(stderr)

As it is covered in How can I split and re-join STDOUT from multiple processes?

Except, command1 outputs different text to both, stdout and stderr. So, it is a combination of the above question and Pipe stderr to another command

To have a context, what I am trying to achieve:

1. Execute curl
2. Capture raw output (stdout), base64, and embed it into json: curl https://someaddress.tld | base64 | jq --raw-input '{"curl_ret" : .}'
3. Output curl json (return code etc) and pass it to stderr: curl --write-out '%{stderr}%{json}' https://someaddress.tld
4. Given #2 and #3 is the same curl call, I want to merge the output of #2 and #3 and pass the merged result to jq: jq --slurp ...

All these in one piped command.

stdout and stderr separation is done to avoid parsing merged text, to avoid pitfalls, given curl output can be anything. Curl has --silent switch so no unexpected text in either output stream

Answer 1

In practice, for the use case at hand

You don't need to do this at all. --write-out '%{json}' will always be written after the body content, so it's always the last line of stdout. It's safe to have it in the same stream.

getExampleAndMetadata() {
  curl --write-out '%{json}' https://example.com |
    jq -RSs '
      split("\n")
      | {"body": .[:-1] | join("\n"),
         "logs": .[-1] | fromjson}'
}

---

As an exercise

It's ugly, and I don't recommend it -- it'd be better to just parse the two sets of data out of stdout and leave stderr free so you have a way to log actual errors -- but, doing what you asked for:

getExampleAndMetadata() {
  local tmpdir retval=0
  # mkfifo is different between operating systems, adjust to fit
  tmpdir=$(mktemp -d sa_query.XXXXXX) || return
  mkfifo "$tmpdir/stdout" "$tmpdir/stderr" || { retval=$?; rm -rf "$tmpdir"; return "$retval"; }
  curl --silent --write-out '%{stderr}%{json}' https://example.com \
    >"$tmpdir/stdout" 2>"$tmpdir/stderr" & curl_pid=$!
  # must read stdout _before_ stderr to avoid a deadlock here
  # to stop caring what order of operations jq uses, we use process substitutions
  # to buffer both stdout and stderr in-memory
  jq -Rn \
    --rawfile content <(out=$(cat "$tmpdir/stdout"); printf '%s\n' "$out") \
    --slurpfile logs <(err=$(cat "$tmpdir/stderr"); printf '%s\n' "$err") \
    '{"body": $content, "logs": $logs}'; (( retval |= $? ))
  rm -rf -- "$tmpdir"; (( retval |= $? ))
  wait "$curl_pid"; (( retval |= $? ))
  return "$retval"
}

...gives you a simple command, getExampleAndMetadata. And of course, if you eliminate the comments and line continuations you can collapse the whole thing to one line by adding ;s appropriately.