4

I want to make an post request using curl. The body is a binary file but I also want curl to encode a url-search parameter.

curl --request POST \
     --header 'Content-Type:application/octet-stream' \
     --data-binary "@file.txt" \ # The body part, raw, not encoded
     --data-urlencode "name=john" \ # The url-search-param part, encoded
     "https://example.com"

The problem is that curl treats --data-urlencode as part of the body, what I want is that name=john gets appended to the url part like this: https://example.com?name=john whereby the --data-binary part gets send over as the POST body. (in reality the search-parameter is a string with "invalid" url-characters which need to be encoded)

TLDR: I want to use --data-urlencode as if I'm making a GET request (to append the parameter) and the POST --data-binary to set the actual POST body.

A quick google search gives me the info to use --get / -G but this transforms the request into a GET request what I don't want so there are already a dozen questions about this on SO (but none of them cover my case):

... and many other duplicates of how to just encode the body or url part.

Simon
  • 2,686
  • 2
  • 31
  • 43

2 Answers2

6

You can't. (until curl 7.87.0 ships with the new --url-query option)

The --data* options are used to either build the request body, or you can ask curl to put them all as query parameters in the URL (with -G). Those options cannot build both, in a single command line.

My advice would be to build the URL "manually" and keep the --data options to build the request body, mostly because the URL part tends to be easier and smaller.

Until curl 7.87.0 is installed on your machine, because once it does you can use --url-query and --data-urlencode and happiness shall ensue.

Daniel Stenberg
  • 54,736
  • 17
  • 146
  • 222
  • Thank you, but what do you mean with "manually", encoding parts of it manually and appending it? I mean, yes, that'd an option and there are already many answers of how to do this but there's no _clean_ way of doing it... – Simon Jun 27 '20 at 11:14
  • @Daniel Stenberg: having `--data-urlencode` and `--data-binary` couldn't be a nice curl enhancement? BTW thanks for building/maintaining curl! – Giorgio Robino May 20 '21 at 08:45
1

curl won't mix post data and query strings for you.

I found a convenient solution for keeping everything in a single bash script without adding additional dependencies. If python, ruby, or node is available, they have url encoding libraries.

Must construct urlencoded query string yourself

A bash only solution for constructing the query string was provided by Chris Down. It is useful gist with an urlencode function that I found suitable. The comments of his gist have modifications for use in other shells.

# From Chris Down https://gist.github.com/cdown/1163649
urlencode() {
    # urlencode <string>
    old_lc_collate=$LC_COLLATE
    LC_COLLATE=C
    local length="${#1}"
    for (( i = 0; i < length; i++ )); do
        local c="${1:$i:1}"
        case $c in
            [a-zA-Z0-9.~_-]) printf '%s' "$c" ;;
            *) printf '%%%02X' "'$c" ;;
        esac
    done
    LC_COLLATE=$old_lc_collate
}

# Construct query string parameter
VAL_1="http://localhost:3030/ds/spek"
ENC_1=$(urlencode "${VAL_1}")
PARAM_1="graph=${ENC_1}"

curl -X PUT --data-binary "@${SPEK_FILE}" \
  --header 'Content-type: application/ld+json' \
  "http://localhost:3030/ds?${PARAM_1}"

# > Content-type: application/ld+json
# > Content-Length: 1215

Using both options appends the data to the body.

Attempting to use both --databinary and --data-encode in a POST ends up with both being appended to the body. Notice the content length difference with the above example.

curl -vvvv -X PUT --data-binary "@${SPEK_FILE}" \
  --header 'Content-type: application/ld+json' \
  --data-urlencode "graph=http://localhost:3030/ds/spek" \
  'http://localhost:3030/ds'

# > Content-type: application/ld+json
# > Content-Length: 1263
GcL
  • 501
  • 6
  • 16