Stack Overflow
Stack Overflow

Questions tagged [google-cloud-python]

Use this tag if you are going to refer something from Python libraries OR API Clients of Google Cloud Platform.

207 questions
39
votes
3 answers

Efficiently write a Pandas dataframe to Google BigQuery

I'm trying to upload a pandas.DataFrame to Google Big Query using the pandas.DataFrame.to_gbq() function documented here. The problem is that to_gbq() takes 2.3 minutes while uploading directly to Google Cloud Storage takes less than a minute. I'm…
Pablo
  • 3,135
  • 4
  • 27
  • 43
38
votes
5 answers

Google Cloud Storage + Python : Any way to list obj in certain folder in GCS?

I'm going to write a Python program to check if a file is in certain folder of my Google Cloud Storage, the basic idea is to get the list of all objects in a folder, a file name list, then check if the file abc.txt is in the file name list. Now the…
Reed_Xia
  • 1,300
  • 3
  • 17
  • 29
22
votes
3 answers

Blob.generate_signed_url() failing to AttributeError

So I'm trying to produce temporary globally readable URLs for my Google Cloud Storage objects using the google-cloud-storage Python library (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) - more specifically the…
Matti
  • 427
  • 1
  • 4
  • 13
19
votes
3 answers

Google Cloud Storage: How to Delete a folder (recursively) in Python

I am trying to delete a folder in GCS and its all content (including sub-directories) with its Python library. Also I understand GCS doesn't really have folders (but prefix?) but I am wondering how I can do that? I tested this code: from…
kee
  • 10,969
  • 24
  • 107
  • 168
18
votes
5 answers

Google Cloud Storage Signed URLs with Google App Engine

It's frustrating to deal with the regular Signed URLs (Query String Authentication) for Google Cloud Storage. Google Cloud Storage Signed URLs Example -> Is this really the only code available in the whole internet for generating Signed URLs for…
roberto.cr
  • 322
  • 3
  • 6
15
votes
1 answer

What is the difference between google.cloud.pubsub_v1 and google.cloud.pubsub?

I see both used in different documentation from Google: from google.cloud import pubsub Is found…
gunit
  • 3,700
  • 4
  • 31
  • 42
13
votes
2 answers

How to use Google Cloud PubSub and Run to handle resource-intensive long-running tasks?

I've got a Google Cloud PubSub topic which at times has thousands of messages and at times zero messages coming in. These messages represent tasks which can take upwards of an hour each. Preferably I'm able to use Cloud Run for this, as it scales…
thimo visser
  • 155
  • 2
  • 7
12
votes
1 answer

How to authenticate with gcloud big query using a json credentials file?

In the gcloud documentation for google bigquery, it states that authentication can be determined from from_service_account_json. I've tried the following from gcloud import bigquery client =…
Chris
  • 5,444
  • 16
  • 63
  • 119
9
votes
3 answers

How to find the current project id of the deployed Python function in Google Cloud gives error

I have deployed a Python 3.7 function in Google Cloud. I need to get the project-id through code to find out where it is deployed. I write a small Python 3.7 script and test it through Google shell command line import urllib import…
user1403505
  • 895
  • 2
  • 19
  • 42
9
votes
3 answers

Set metadata in Google Cloud Storage using gcloud-python

I am trying to upload a file to Google Cloud Storage using gcloud-python and set some custom metadata properties. To try this I have created a simple script. import os from gcloud import storage client = storage.Client('super secret app…
Remco Haszing
  • 7,178
  • 4
  • 40
  • 83
8
votes
1 answer

Mock the result of accessing public GCS bucket

I have the following code: bucket = get_bucket('bucket-name') blob = bucket.blob(os.path.join(*pieces)) blob.upload_from_string('test') blob.make_public() result = blob.public_url # result is `
Rodrigo
  • 135
  • 4
  • 45
  • 107
8
votes
2 answers

Retrieve list of log names from Google Cloud Stackdriver API with Python

I'm using Google's Stackdriver Logging Client Libraries for Python to programmatically retrieve log entries, similar to using gcloud beta logging read. Stackdriver also does provide an API to retrieve a list of log names, which is most probably what…
otto.poellath
  • 4,129
  • 6
  • 45
  • 60
7
votes
1 answer

Increase Connection Pool Size

We are running the following code to upload to GCP Buckets in parallel. It seems we are quickly using up all the connections in the pool based on the warnings we are seeing. Is there any way to configure the connection pool the library is using? def…
Michael
  • 679
  • 7
  • 24
7
votes
2 answers

google-cloud-storage distribution not found despite being installed in venv

As in the title, when running the appserver I get a DistributionNotFound exception for google-cloud-storage: File "/home/[me]/Desktop/apollo/lib/pkg_resources/init.py", line 867, in resolve raise DistributionNotFound(req, requirers) …
J Shingle
  • 71
  • 3
7
votes
3 answers

Load schema json file to create table or job schema

If I already have the schema file, for example: schema.json. How can I load the file to create the table or job schema using the google-cloud-python API?
Tuan Vu
  • 708
  • 7
  • 15
1
2 3
…
13 14