6

I'm launching container images on Google Cloud AI Training (Cloud ML Engine)

Inside those containers I need to use gsutil. Some containers have gsutil. In that case I can use it right away without any authentication steps.

Some containers do not have gsutil, so I have to install it. The problem is that the installed gsutil does not work.

When I'm using the official cloud-sdk image, gsutil works without any auth steps.

When I use the python:3.7 image and install gsutil from PyPI it does not work:

python -m pip install gsutil --quiet
gsutil cp a gs://b/c

ServiceException: 401 Anonymous caller does not have storage.objects.get access to ...

How can I make it so that the standalone gsutil obtains the needed credentials?

Most guides focus on manually calling gcloud auth, copying URL and copying back the token. This is not the solution that I seek (which should be automated). I know that the automated solution is possible since in some images gsutil works out of the box.

Ark-kun
  • 6,358
  • 2
  • 34
  • 70
  • Can you try this: https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/training/horovod/base/Dockerfile – gogasca Apr 24 '20 at 04:40

1 Answers1

10

This is because that pip install gsutil alone does not configure the credentials, which is why it's anonymous user as the error says. You'll want to configure credentials to access protected data.

Put following line in your docker file and it should work:

RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg

It's to configure gsutil to use the default service account.

Bo yang
  • 136
  • 1
  • 3
  • Fantastic answer! This in fact allows `gsutil` to use the service account in other GCP instances, such as Compute Engine's Container-Optimised OS. Using your answer I created a lighteight wrapper image for `gsutil` intended for GCP instances, [check it out on GitHub](https://github.com/Voyz/gsutil_wrap). I wonder, where did you read about this solution? Can't find anything in the docs. Massive thanks either way! – Voy Oct 29 '20 at 05:23
  • this worked for me partially, let me explain i have two projects, first one worked with solution provided, but second one no, for each one i have a service account, this also can work: RUN echo '[GoogleCompute]\nservice_account = serv-acc-name@xxx-xx-xx.iam.gserviceaccount.com' > /etc/boto.cfg you can change "default" for the name of your service account name. – Pavul Zavala Jul 15 '21 at 04:10
  • 1
    Do you happen to know how to achieve the same thing for local user credentials. Where did you find information about this? – NickTheDev Mar 28 '22 at 19:22
  • Also interested about user credentials.. – Patrick Geyer Sep 16 '22 at 18:02
  • Turns out \n wasn't creating the new line with the original command. This one with `$` interpolation worked `RUN echo $'[GoogleCompute]\nservice_account = default' > /etc/boto.cfg` – vozman Mar 30 '23 at 09:59