0

I have a bash function to repeated curl call, accepting different URLs as function args. The urls are different and one of them has previous query, like "?foo=bar". The API behind these urls are defined by users but they meet the same standard and need a mandatory arg like client=xxx. So I am thinking how to do this in bash scripting.

Now I am doing:

    client_query=client=xxx
    oauth_url_with_client=$(echo $oauth_url | sed 's/?/?'$client_query'&/g') # if found, replace first query with client query
    echo "First step: $oauth_url_with_client"
    # when there is no "?", no replace happens, need to append
    if [[ $oauth_url_with_client != *"$client_query"* ]]; then
        oauth_url_with_client=$oauth_url_with_client?$client_query
    fi
    echo "Final oauth url: $oauth_url_with_client"

But now I see:

First step: https://host/oauth2/token?client=xxx?foo=bar
Final oauth url: https://host/oauth2/token?client=xxx?foo=bar

I see sed is not replacing ? with ?client=xxx&. What is the problem here?

Is there some good way to combine possible/optional query from URL with mandatory query in --data or --form? Not sure where to put these

WesternGun
  • 11,303
  • 6
  • 88
  • 157
  • 2
    The lack of quoting seems like a possible explanation. Try https://shellcheck.net/ before asking for human assistance. See also [When to wrap quotes around a shell variable](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – tripleee Mar 17 '23 at 08:37
  • 1
    @tripleee : The only potentially risky (i.e. unquoted) places are `$oauth_url` and `$client_query`, and perhaps the second assignment to `oauth_url_with_client`. We don't know the content of `oauth_url`, but even if it contains a glob character, this would most likely be a `?` and it would be unlikely that there is a matching filename which gets expanded here. I think running the script with `set -x` would be informative. – user1934428 Mar 17 '23 at 09:06
  • `sed -i` is definitely wrong here, but I don't think that's the explanation for the problem, either. Tangentially, the `g` flag seems like unnecessary carbon cult copy, too. – tripleee Mar 17 '23 at 09:09
  • Hm.. thanks for the suggestions. I am confused why `sed` not working, but shellcheck.net did give me one advice: `consider using ${var//search/replace}` and that works for me with some quoting. But the purpose is kind of making sure that I didn't miss anything about what curl can do; just in case I am making some x-y problem mistake. – WesternGun Mar 17 '23 at 09:10
  • sorry about `-i`; need to be removed. – WesternGun Mar 17 '23 at 09:11
  • I add quotes around `$client_query` in `sed` but that does not work, still not appending `&`, but `?`: `oauth_url_with_client=$(echo $oauth_url | sed 's/?/?'"$client_query"'&/g')` – WesternGun Mar 17 '23 at 09:17
  • 1
    OK... `&` seems to be special in `sed` pattern; needs escaping `\&`. https://unix.stackexchange.com/questions/296705/using-sed-with-ampersand – WesternGun Mar 17 '23 at 09:19

1 Answers1

0

Thanks for the suggestions from @tripleee, that I could check with shellcheck.net, which gave me a suggestion to use ${var//search/replace} form instead of sed; then I came up with this which works:

oauth_url_with_client=${oauth_url//"?"/"?$client_query&"}

Here I am not afraid of quoting; in the sed pattern I fear there would be side effects but here I can quote as much as I like.

But I also agree with @user1934428 that actually no expanding should happen in an url; I have considered the possible sequence of not quoting actually. And, quoting does not solve my problem.

And, after that, I am thinking if & should be treated specially and a search of "sed & pattern" leads me to https://unix.stackexchange.com/questions/296705/using-sed-with-ampersand which confirms me that & represents found part in search so it needs escaping:

oauth_url_with_client=$(echo $oauth_url | sed 's/?/?'"$client_query"'\&/g')

which also works.

The last question still stands: if curl could it natively? I will leave it to the future since my problem is solved in two ways already; but I guess this is too much for curl, it does not expect mixing queries from url and from options.

WesternGun
  • 11,303
  • 6
  • 88
  • 157