How do I upload a big file using Python 3 using only built-in modules?

Question

I'm trying to upload a file 32+ MB to a server using an API. The only restriction I have is that can only use built-in modules. I have seen many examples using requests library, but I'm trying to solve with urllib. Using curl as PoC, I did the job like this:

curl -v --request POST --url 'https://domain/upload/long-string/' --form 'apikey=my-api-key' --form 'file=@my-file.extension'

Using urllib, I wrote the code below, but it doesn't work, because the server always returns a 400 error:

import urllib

def post_bigfile(upload_url, file, auth, timeout):
        headers = {'Accept': '*/*', 'Content-Type': 'multipart/form-data'}
        data = {'file': file, 'apikey': auth}
        req = urllib.request.Request(upload_url, headers=headers, 
            data=urlencode(data).encode('utf-8'), method='POST')
        return urllib.request.urlopen(req, timeout=timeout)

post_bigfile('https://domain/upload/long-string/', open('my-file.extension','rb'), 'my-api-key', 20)

I have tried using different values of Content-Type and Accept, but it still doesn't work. What could I possibly doing wrong? Is there another built-in module I could use to better solve this problem?

What exactly is `file` here... is it a filename, a file like object, the content of a file? You might also need to do some manual wrangling to construct a suitable multipart/form-data body so the file data can be recognised as such and separated from the "apikey" (you might want to look at the `requests` library and how it creates the request... mind you... you'll probably keep coming across issues trying to do anything even slightly non-straightforward purely using `urllib` - so it's worth arguing that using `requests` will probably save plenty of time and hassle and potential bugs...) — Jon Clements, May 03 '19 at 12:38
I call that function like this: `post_bigfile('https://domain/upload/long-string/', open('my-file.extension','rb'), 'my-api-key', 20)`. Since I need to use built-in functions, `requests` is not an option. I have successfully uploaded a file using `curl`. Using the same file with my function, it doesn't work, so I'm missing something. — forkd, May 03 '19 at 13:33

cardamom · Answer 1 · 2019-05-03T14:21:14.360

A hint to what you are doing wrong can be found here:

'Content-Type': 'multipart/form-data'
headers=headers

I was also busy with curl commands a while ago which were pushing multipart form-data and investigating doing it with built-in python libraries only (ie not requests). Once you put --form into curl you don't even need to give curl the --headers "Content-Type: multipart/form-data" argument, it just defaults to it. Maybe you didn't realise curl was doing that.

Have a look at this, python does not support that Mime type. A Python issue is linked https://bugs.python.org/issue3244 and I think your best bet is this script: https://pymotw.com/3/urllib.request/#uploading-files if you really want to do that with standard libraries only.

Am assuming you don't control the server which is receiving that file and it looks like it wants multipart/form-data with the two elements "apikey" and "file". Unless you can alter what that server wants to something which urllib can POST you will have to use requests or see if you can get that big MultiPartForm class to work.

How do I upload a big file using Python 3 using only built-in modules?

1 Answers1