Python Requests Messing with Links in my Data Dict

Question

I've got a request that looks like this:

data = {
    'USER': params['PARTNER'],
    'PWD': params['PWD'],
    'RETURNURL': 'https://my-site.com/',
    'CURRENCY': 'USD',
}

request = requests.post(url=URL, data=data, verify=False)

When I run this command in an identical cURL command in the shell, there's no problem at all. But when I do it through requests, it seems like something gets messed up with the RETURNURL parameter. It's preserved perfectly in cURL, but when processed from requests, it ends up looking like this when I see it on the other side of the request and things just don't work:

https%3A%2F%2Fmy-site.com%2F

Thanks in advance, been at this forever!

The actual command being used successfully in cURL is this:

curl https://pilot-payflowpro.paypal.com \
  -s \
  --insecure \
  -d PARTNER=PayPal \
  -d PWD=MyPassword \
  -d VENDOR=MyMerchantID \
  -d USER=MyMerchantID \
  -d TENDER=C \
  -d ACCT=5105105105105100 \
  -d TRXTYPE=S \
  -d EXPDATE=1221 \
  -d RETURNURL=https://my-site.com/
  -d AMT=1.00

Note that everything is working perfectly, but the parameter I submit which is a URL is the one that simply won't work and the formatting is all messsed up.

Show the curl command you are using. And what is processing the data on the other side? Is it an application you control? — Daniel Roseman, Nov 21 '18 at 09:58
@DanielRoseman I'm accessing a PayPal API, will update the post with some more info in 30 seconds — capcom-r, Nov 21 '18 at 10:01
OK but how do you know how the URL is being received by PayPal? — Daniel Roseman, Nov 21 '18 at 10:05
@DanielRoseman Well the RETURNURL parameter is not necessary, so if I take it out everything works fine. But I need that parameter in there to customize the functionality for my use-case. — capcom-r, Nov 21 '18 at 10:07
Similar to this question here - https://stackoverflow.com/questions/23496750/how-to-prevent-python-requests-from-percent-encoding-my-urls — Mortz, Nov 21 '18 at 10:09

score 0 · Answer 1 · answered Nov 21 '18 at 10:00

0

Assuming you're sending the data as JSON, then send the data with the json parameter instead of data

request = requests.post(url=URL, json=data, verify=False)

answered Nov 21 '18 at 10:00

AdrienF

859
1
6
19

Added some more detail about the cURL command in the original post for reference. – capcom-r Nov 21 '18 at 10:04
I would double-check that; JSON string values don't get processed (which you can verify by calling `json.dumps` on your dictionary). I just run both cases locally, using Charles proxy to check the payload. The JSON payload is sent unmodified, the string payload is URL-encoded. – AdrienF Nov 21 '18 at 10:16

score 0 · Answer 2 · answered Nov 22 '18 at 05:13

Use --trace-ascii log.txt to log down curl post process.

curl

=> Send data, 161 bytes (0xa1)
0000: PARTNER=PayPal&PWD=MyPassword&VENDOR=MyMerchantID&USER=MyMerchan
0040: tID&TENDER=C&ACCT=5105105105105100&TRXTYPE=S&EXPDATE=1221&RETURN
0080: URL=https://my-site.com/&AMT=1.00

requests with data = data_dict

Content-Length: 169

ACCT=5105105105105100&TENDER=C&EXPDATE=1221&PARTNER=PayPal&RETURNURL=https%3A%2F%2Fmy-site.com%2F&VENDOR=MyMerchantID&USER=MyMerchantID&TRXTYPE=S&AMT=1.00&PWD=MyPassword

So you can try to post your data as string, as @Mortz said.

import requests

data = {
    'PARTNER':'PayPal',
    'VENDOR':'MyMerchantID',
    'TENDER':'C',
    'ACCT':'5105105105105100',
    'TRXTYPE':'S',
    'EXPDATE':'1221',
    'AMT':'1.00',
    'USER': 'MyMerchantID',
    'PWD': 'MyPassword',
    'RETURNURL': 'https://my-site.com/',
}
data = "&".join("{}={}".format(k,v)  for k,v in data.items())
print(len(data), data)

url = "https://pilot-payflowpro.paypal.com"

request = requests.post(url=URL, data=data, verify=False)

Python Requests Messing with Links in my Data Dict

2 Answers2