How to upload a file to Google Cloud Storage on Python 3?

Question

How can I upload a file to Google Cloud Storage from Python 3? Eventually Python 2, if it's infeasible from Python 3.

I've looked and looked, but haven't found a solution that actually works. I tried boto, but when I try to generate the necessary .boto file through gsutil config -e, it keeps saying that I need to configure authentication through gcloud auth login. However, I have done the latter a number of times, without it helping.

Answer 1

Use the standard gcloud library, which supports both Python 2 and Python 3.

Example of Uploading File to Cloud Storage

from gcloud import storage
from oauth2client.service_account import ServiceAccountCredentials
import os


credentials_dict = {
    'type': 'service_account',
    'client_id': os.environ['BACKUP_CLIENT_ID'],
    'client_email': os.environ['BACKUP_CLIENT_EMAIL'],
    'private_key_id': os.environ['BACKUP_PRIVATE_KEY_ID'],
    'private_key': os.environ['BACKUP_PRIVATE_KEY'],
}
credentials = ServiceAccountCredentials.from_json_keyfile_dict(
    credentials_dict
)
client = storage.Client(credentials=credentials, project='myproject')
bucket = client.get_bucket('mybucket')
blob = bucket.blob('myfile')
blob.upload_from_filename('myfile')

Answer 2

A simple function to upload files to a gcloud bucket.

from google.cloud import storage
#pip install --upgrade google-cloud-storage. 
def upload_to_bucket(blob_name, path_to_file, bucket_name):
    """ Upload data to a bucket"""
     
    # Explicitly use service account credentials by specifying the private key
    # file.
    storage_client = storage.Client.from_service_account_json(
        'creds.json')

    #print(buckets = list(storage_client.list_buckets())

    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(blob_name)
    blob.upload_from_filename(path_to_file)
    
    #returns a public url
    return blob.public_url

You can generate a credential file using this link: https://cloud.google.com/storage/docs/reference/libraries?authuser=1#client-libraries-install-python

Asynchronous Example:

import asyncio
import aiohttp
# pip install aiofile
from aiofile import AIOFile
# pip install gcloud-aio-storage
from gcloud.aio.storage import Storage 

BUCKET_NAME = '<bucket_name>'
FILE_NAME  = 'requirements.txt'
async def async_upload_to_bucket(blob_name, file_obj, folder='uploads'):
    """ Upload csv files to bucket. """
    async with aiohttp.ClientSession() as session:
        storage = Storage(service_file='./creds.json', session=session) 
        status = await storage.upload(BUCKET_NAME, f'{folder}/{blob_name}', file_obj)
        #info of the uploaded file
        # print(status)
        return status['selfLink']
        

async def main():
    async with AIOFile(FILE_NAME, mode='r') as afp:
        f = await afp.read()
        url = await async_upload_to_bucket(FILE_NAME, f)
        print(url)


# Python 3.6
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

# Python 3.7+
# asyncio.run(main()) 

Answer 3

Imports the Google Cloud client library (need credentials)

from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="C:/Users/siva/Downloads/My First Project-e2d95d910f92.json" 

Instantiates a client

storage_client = storage.Client()

buckets = list(storage_client.list_buckets())

bucket = storage_client.get_bucket("ad_documents") # your bucket name

blob = bucket.blob('chosen-path-to-object/{name-of-object}')
blob.upload_from_filename('D:/Download/02-06-53.pdf')
print(buckets)

Answer 4

When installing Google Cloud Storage API:

pip install google-cloud

will throw a ModuleNotFoundError:

    from google.cloud import storage
ModuleNotFoundError: No module named 'google'

Make sure you install as in Cloud Storage Client Libraries Docs:

pip install --upgrade google-cloud-storage

Answer 5

This official repo contains a handful of snippets demonstrating the different ways to upload a file to a bucket: https://github.com/googleapis/python-storage/tree/05e07f248fc010d7a1b24109025e9230cb2a7259/samples/snippets

• upload_from_string()
• upload_from_file()
• upload_from_filename()