Python requests POST method parses XML always as ANSI

Question

I have a script which sends a XML within a POST method using python requests module to an API. This code works from my test machine (windows) when I deploy the script to my PROD server (Debian) I always receive a parsing error. If I change the encoding of the file to ANSI it will work on the Debian server.

It seems to be some kind of encoding issue. However from my point of view I set up everything correctly in order to send the file as utf-8.

This is the relevant code snippet:

# prepare XML for API call
    url = "https://myurl.com/api/method"
    # use XML template and replace some wildcards
    payloadTemplate = open("message.xml", encoding="utf-8")
    payload = payloadTemplate.read()
    payload = str(payload % (sessiontoken, accountName, key, userId))
    headers = {
        'X-XYZ-API-COMPATIBILITY-LEVEL': '1234',
        'X-XYZ-API-CALL-NAME': 'methodname',
        'X-XYZ-API-SITEID': '11',
        'Content-Type': 'application/xml; charset=utf-8'
    }
    # send request
    response = requests.request("POST", url, headers=headers, data=payload)

The XML looks like this:

<?xml version="1.0" encoding="utf-8"?>
<APIMethod xmlns="urn:Tool:apis:eBLBaseComponents">
<RequesterCredentials>
<ToolAuthToken>%s</ToolAuthToken>
</RequesterCredentials>
<ItemID>12345678</ItemID>
<MemberMessage>
<Subject>Thanks</Subject>
<Body>random text some information %s another information %s</Body>
<QuestionType>General</QuestionType>
<RecipientID>%s</RecipientID>
</MemberMessage>
</APIMethod>

Using Postman or the script on Windows is working fine. On Debian it's not working anymore. If I save the XML file in ANSI encoding it will work on Debian as well. I'm using this as workaround, unfortuantely I would like to have the same solutions for Windows and Unix (which is in my case TEST and PROD). Also I would like to include proper German letters.

Glad for any help

I added the charset=utf-8 to the header. It changes nothing un Debian side. In windows its still working. When I'm printing the payload as string, everything is fine. There seems to be no encoding issue.

The default header are different in Postman from other applications. On working system check default http headers that you are not setting. Then add the same settings to your code. — jdweng, Aug 01 '23 at 12:10
Have you verified that the read XML is really encoded using UTF-8? // If TRUE a hexdump of the file should show German Umlaute 2-byte encoded, always containing the byte `c3`: `hexdump -C <<< äöüßÄÖÜ` => `00000000 c3 a4 c3 b6 c3 bc c3 9f c3 84 c3 96 c3 9c 0a |...............|` — dodrg, Aug 01 '23 at 12:58
Yes, it's encoded UTF8 I checked that. @jdweng it's working on windows, therefore I guess I did not miss any headers. — Sven Sven, Aug 01 '23 at 13:06
I said check ALL headers. The language header or the userAgent could be different. — jdweng, Aug 01 '23 at 13:13
Also basic check: The UTF-8 does *not* contain a [BOM](https://stackoverflow.com/questions/2223882/whats-the-difference-between-utf-8-and-utf-8-with-bom) (does not start wih the binary values `EF BB BF`)? — dodrg, Aug 01 '23 at 13:13
@jdweng Sorry, I did not understand. How can the headers affect the encoding, when there are the same on Windows and unix. Windows works Unix not. The headers should affect it on both systems, if they would be the source of troubles. — Sven Sven, Aug 01 '23 at 13:42
I just checked on another system, which has Linux Mint in place. It also works on here. Therefore I suspect a unique issue with Debian. — Sven Sven, Aug 01 '23 at 14:36
When a client connects to a server there is a negotiation that occurs using http headers to find common options for transfer. When client doesn't send an option the default option of server is used. — jdweng, Aug 01 '23 at 17:46
Please post sample of your XML, redact as needed, and any specific errors you receive. Very interested to see how you are string formatting with the modulo operator, `%`, on XML content. Consider parsing XML with a conformant DOM library like etree or lxml. — Parfait, Aug 02 '23 at 01:49
By the way, the `%` for string formatting is now no longer recommended and [de-emphasized in Python](https://stackoverflow.com/a/13452357/1422451) (not deprecated *yet*). Please also relay the Python versions on either machine. — Parfait, Aug 02 '23 at 01:50
I posted the XML content in the initial question. Python version are somewhat idential. Debian is: 3.11.2 windows is: 3.11.2 Linux Mint is: 3.11.4 — Sven Sven, Aug 02 '23 at 15:31
Ok. Great. One last item. *It doesn't work* is not a meaningful description. Please [edit] describing what specifically happens. Does `requests` raise an error? Print out `payload` after string formatting and compare across OSes. Head scratcher why this code block would differ in behavior unless a server related difference is reason. — Parfait, Aug 03 '23 at 01:04
As stated the API responds with a XML parsing error. I also tried once to send a prebuilt XML without the string replacement, still had the parser error. I guess I just have to live with it. — Sven Sven, Aug 03 '23 at 07:24
It says the openening tag of APImethod was not closed. It is some Debian issue. I will setup another debian system if I find the time and let you know if this was a local issue. — Sven Sven, Aug 03 '23 at 15:43

score 0 · Answer 1 · answered Aug 01 '23 at 14:51

0

Check the locale at your Debian system.

Verify you have installed and setup the right locale.

For the current settings:

locale

Configured:

grep -v "^#" /etc/locale.gen

If a relevant locale is missing, edit the file /etc/locale.gen and activate the required entries by deleting the # mark in its line.

Even if configured right, please issue

sudo locale-gen

to be sure they really have been generated.

answered Aug 01 '23 at 14:51

dodrg

1,142
2
18

Thanks for the feedback. Locale is setup as de_DE.UTF-8, which seems to be fine – Sven Sven Aug 01 '23 at 17:22

score 0 · Answer 2 · answered Aug 03 '23 at 17:35

Consider parsing the payload XML template with a DOM library such as Python's built-in xml.etree and use library to update element text. Avoid treating the XML like a text file with str conversion and formatting on entire content. For string formatting of text, use the current recommended standard of F-strings:

import xml.etree.ElementTree as et
...

# handle default namespace
et.register_namespace('', "urn:Tool:apis:eBLBaseComponents")
nmsp = {"doc": "urn:Tool:apis:eBLBaseComponents"}

# parse XML template and replace element text
msg = et.parse("XMLAPIPayload.xml")

msg.find(".//doc:RequesterCredentials/doc:ToolAuthToken", namespaces = nmsp).text = sessiontoken
msg.find(".//doc:MemberMessage/doc:Body", namespaces = nmsp).text = (
    f"random text some information {accountName} another information {key}"
)
msg.find(".//doc:MemberMessage/doc:RecipientID", namespaces = nmsp).text = userId

# export to byte string
payload = et.tostring(msg.getroot(), encoding="utf-8")

# prepare XML for API call
url = "https://myurl.com/api/method"

headers = {
    'X-XYZ-API-COMPATIBILITY-LEVEL': '1234',
    'X-XYZ-API-CALL-NAME': 'methodname',
    'X-XYZ-API-SITEID': '11',
    'Content-Type': 'application/xml; charset=utf-8'
}

# send request
print(payload)
response = requests.request("POST", url, headers=headers, data=payload)

Thanks for the effort. Somehow I manage to get the code to run, I was receiving a lot of errors in the first place. Now it works, nevertheless on Debian I receive the parser error from the API as response. — Sven Sven, Aug 04 '23 at 15:40

score 0 · Accepted Answer · answered Aug 09 '23 at 08:38

0

After a fresh install of Debian the issue no longer exists. Python version remained the same. Therefore, I assume (like I did in the beginning) that this was some kind of Debian caused issue.

answered Aug 09 '23 at 08:38

Sven Sven

1
2

Python requests POST method parses XML always as ANSI

3 Answers3