How to parse POST data into a dictionary

Question

I'm trying to write a simple "server" that inherits from BaseHTTPRequestHandler and is able to receive GET and POST requests. I have a server.py file that, so far, looks like this:

from http.server import BaseHTTPRequestHandler, HTTPServer
from cgi import parse_header, parse_multipart
from urllib.parse import parse_qs

host = "localhost"
port = 8080

hello_msg = "Server running..."

class Server(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        self.respond_OK(hello_msg)

    def do_POST(self):
        print("Post")

        data = self.parse_POST()

        print(data)
        print(type(data))

        self.respond_OK("End post")


    def parse_POST(self):
        ctype, pdict = parse_header(self.headers['content-type'])
        if ctype == 'multipart/form-data':
            postvars = parse_multipart(self.rfile, pdict)
        elif ctype == 'application/x-www-form-urlencoded':
            length = int(self.headers['content-length'])
            postvars = parse_qs(
                    self.rfile.read(length), 
                    keep_blank_values=1)
        else:
            postvars = {}

    
        return postvars

    def respond_OK(self, msg):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        self.wfile.write(bytes(msg, "utf-8"))


if __name__ == "__main__":        
    webServer = HTTPServer((host, port), Server)
    print("Server started http://%s:%s" % (host, port))

    try:
        webServer.serve_forever()
    except KeyboardInterrupt:
        pass

    webServer.server_close()
    print("Server stopped.")

And a client.py file that looks like this:

import requests

sample = {
    "msg": "sample post message",
    "number": 159687,
    "list": ["one", "two", "three"]
}

url = "http://localhost:8080"

r = requests.post(url, sample)

In the do_POST function, I want the data variable to be exactly the same as the sample variable that's being sent by the client, but what I end up with is this:

{b'msg': [b'sample post message'], b'number': [b'159687'], b'list': [b'one', b'two', b'three']}

As you can see, it is a dictionary, but the keys and values are bytes and not strings. I could simply convert the keys and values after parsing them, but I have the impression that I'm doing something wrong, and that the result of the parsing should already be strings instead of bytes.

What am I doing wrong? Is the solution here to simply add another step where I convert bytes to string?

first you could check what you get with `self.rfile.read(length)` - maybe it need different method to encoding. But I think all is OK and you have to convert it to string on your own. If you decided to use `http.server` instead of web frameworks like `Flask` then you have to write all code on your own. — furas, Jun 16 '21 at 23:15
The output of `self.rfile.read(length)` is `b'msg=sample+post+message&number=159687&list=one&list=two&list=three'`, which I guess is URL encoding. As for the last part of your comment, it makes sense. Frameworks exist for a reason after all. Thanks. — Tuma, Jun 16 '21 at 23:24
if you use `.decode()` then you can create strings instead of bytes - `parse_qs( self.rfile.read(length).decode(), ... )` - but lists you will have to convert on your own. — furas, Jun 17 '21 at 00:41

score 5 · Accepted Answer · answered Jun 17 '21 at 00:47

If you use .decode() then you get strings instead of bytes.

result = parse_qs(data.decode(), strict_parsing=True)

But lists you have to convert on your own.

for key in result:
    if len(result[key]) == 1:         # if list has only one element
        result[key] = result[key][0]  # then get it from list

I use only key (without value) from dictionary to make changes on original dictionary result.

Minimal working example

from urllib.parse import parse_qs

data = b'msg=sample+post+message&number=159687&list=one&list=two&list=three'
data = data.decode()

result = parse_qs(data, strict_parsing=True)
print(result)

for key in result:
    if len(result[key]) == 1:
        result[key] = result[key][0]

print(result)

How to parse POST data into a dictionary

1 Answers1