Google cloud storage signed url chunked download In python?

Question

I'd like to do the following functionality:

  service = get_service('storage', 'v1', auth)

  data = BytesIO()
  request = service.objects().get_media(bucket=bucket, object=filename)
  media = MediaIoBaseDownload(data, request, chunksize=chunksize)

But with a signed URL instead if a bucket and object. How do I chunk download a signed url in python?

Answer 1

You can create s Signed URL either using gsutil or writing your own code to generate it. I have tested it using this:

gsutil signurl -d 10m my-key-file.json gs://my-bucket-name/my-file

After that the problem reduces to:

import urllib2
response = urllib2.urlopen('https://storage.googleapis.com/etc...')
html = response.read()

Or maybe:

from six.moves import urllib
urllib.request.urlretrieve("https://storage.goo.../etc","my-file")
print("File downloaded")

as explained in this answer.

There's also a repo where they have developed code to generate signed urls using python.

Answer 2

Signed URLs are just regular downloads, no need for special cloud APIS, and can be downloaded using a urllib read function answered here:

Stream large binary files with urllib2 to file