Passing a URL with brackets to curl

Question

If I try to pass a URL to curl that contains brackets, it fails with an error:

$ curl 'http://www.google.com/?TEST[]=1'
curl: (3) [globbing] illegal character in range specification at pos 29

However, if I escape both brackets, it appears to work:

$ curl 'http://www.google.com/?TEST\[\]=1'

How do I fix this? Is there an argument that will escape URLs automatically, or a description of the characters that need to be escaped before passing to curl?

Answer 1

Add -g to your command:

-g, --globoff
      This option switches off the "URL globbing parser". When you set this option, you can
      specify URLs that contain the letters {}[] without having curl itself interpret them.
      Note that these letters are not normal legal URL contents but they should be encoded
      according to the URI standard.

      Example:
       curl -g "https://example.com/{[]}}}}"

curl.se/docs/manpage.html#-g

Answer 2

Globbing uses brackets, hence the need to escape them with a slash \. Alternatively, the following command-line switch will disable globbing:

--globoff (or the short-option version: -g)

Ex:

curl --globoff https://www.google.com?test[]=1

Answer 3

None of the above answers worked for me, I have to replace all opening/closing brackets with %5B and %5D.

[ ---> %5B and for

] ---> %5D

My initial curl url was like this this

https://test.com/computer/agent1/api/json?pretty=true&tree=executors[currentExecutable[url]]

Now I am using like this

https://test.com/computer/agent1/api/json?pretty=true&tree=executors%5BcurrentExecutable%5Burl%5D%5D

Answer 4

I was getting this error though there were no (obvious) brackets in my URL, and in my situation the --globoff command will not solve the issue.

For example (doing this on on mac in iTerm2):

for endpoint in $(grep some_string output.txt); do curl "http://1.2.3.4/api/v1/${endpoint}" ; done

I have grep aliased to "grep --color=always". As a result, the above command will result in this error, with some_string highlighted in whatever colour you have grep set to:

curl: (3) bad range in URL position 31:
http://1.2.3.4/api/v1/lalalasome_stringlalala

The terminal was transparently translating the [colour\codes]some_string[colour\codes] into the expected no-special-characters URL when viewed in terminal, but behind the scenes the colour codes were being sent in the URL passed to curl, resulting in brackets in your URL.

Solution is to not use match highlighting.

Answer 5

In the documentation it is written:

globbing letters

The curl command line tool supports "globbing" of URLs. It means that you can create ranges and lists using [N-M] and {one,two,three} sequences. The letters used for this ([]{}) are reserved in RFC 3986 and can therefore not legitimately be part of such a URL.

They are however not reserved or special in the WHATWG specification, so globbing can mess up such URLs. Globbing can be turned off for such occasions (using --globoff).

It means that you should do percent-encoding for reserved/special characters (:/?#[]@!$&'()*+,;=) to avoid their special interpretation. To do so you put percent sign (%) and hex value of character in ASCII table. For example:

Symbol	Encoded Value
[	%5B
]	%5D
{	%7B
}	%7D

curl doesn't expect reserved/special characters in URL and these four symbols are used to generate multiple URLs (globbing operation):

$ curl http://localhost:8080/?TEST[a-c]=1

will equivalent to

$ curl http://localhost:8080/?TESTa=1
$ curl http://localhost:8080/?TESTb=1
$ curl http://localhost:8080/?TESTc=1

and

$ curl http://localhost:8080/?TEST{a,c,e}=1

will equivalent to

$ curl http://localhost:8080/?TESTa=1
$ curl http://localhost:8080/?TESTc=1
$ curl http://localhost:8080/?TESTe=1

If you want to disable globbing operation:

• encode them:

  $ curl http://localhost:8080/?TEST%5Ba-c%5D=1
  $ curl http://localhost:8080/?TEST%7Ba,c,e%7d=1
  

  For zsh (default shell on Mac OS X) you must escape ? as well. Thus for both bash and zsh shells:

  $ curl http://localhost:8080/\?TEST%5Ba-c%5D=1
  $ curl http://localhost:8080/\?TEST%7Ba,c,e%7d=1
  

• or use -g/--globoff option:

  $ curl -g http://localhost:8080/?TEST[a-c]=1
  $ curl -g http://localhost:8080/?TEST{a,c,e}=1  # not enough, see note below
  

  ☝ In last example there is a caveat: globbing may be done by bash and zsh shell. To avoid globbing by shell:

  • either escape characters putting reverse slash (\) (don't forget about escaping ? for zsh shell):

    $ curl -g http://localhost:8080/\?TEST\[a-c\]=1
    $ curl -g http://localhost:8080/\?TEST\{a,c,e\}=1
    

  • or put URL in quotes (single or double):

    $ curl -g 'http://localhost:8080/?TEST[a-c]=1'
    $ curl -g 'http://localhost:8080/?TEST{a,c,e}=1'
    

☝ Also be aware that empty square brackets ([]) don't lead to globbing in curl:

$ curl 'http://localhost:8080/?TEST[]=1'

will request /?TEST[]=1.

This is not true for empty curly brackets ({}):

$ curl 'http://localhost:8080/?TEST{}=1'
curl: (3) empty string within braces in URL position 29:
http://localhost:8080/?TEST{}=1
                            ^

they must contain at least one string.

---

P.S. You may test on docker (press Ctrl+C to quit):

$ docker run --rm -p 8080:80 -it nginx

and run curl against it in separate terminal:

$ curl http://localhost:8080/?TEST[a-c]=1

In logs you should see generate URL for request:

172.17.0.1 - - [17/Jan/2023:09:21:53 +0000] "GET /?TESTa=1 HTTP/1.1" 200 615 "-" "curl/7.86.0" "-"
172.17.0.1 - - [17/Jan/2023:09:21:53 +0000] "GET /?TESTb=1 HTTP/1.1" 200 615 "-" "curl/7.86.0" "-"
172.17.0.1 - - [17/Jan/2023:09:21:53 +0000] "GET /?TESTc=1 HTTP/1.1" 200 615 "-" "curl/7.86.0" "-"