Get compute metadata from app engine, got 404 page not found error

Question

I try to get project-wide metadata from app engine, the url like this:

http://metadata.google.internal/computeMetadata/v1/project/attributes/IT_EBOOKS_API

The stackdriver logging give me an error:

message: '404 - "404 page not found\\n"',

But I can get metadata from compute engine. Here is the metadata output:

novaline_dulin@test:~$ curl http://metadata.google.internal/computeMetadata/v1/project/attributes/IT_EBOOKS_API -H 
"Metadata-Flavor: Google"
http://it-ebooks-api.info/v1novaline_dulin@test:~$

And, here is my code for getting custom project-wide metadata

const request = require('request-promise');

async function getMetaData(attr) {
  const url = `http://metadata.google.internal/computeMetadata/v1/project/attributes/${attr}`;
  const options = {
    headers: {
      'Metadata-Flavor': 'Google'
    }
  };
  console.log('url:', url);
  return request(url, options)
    .then((response) => {
      console.info(`Retrieve meta data successfully. meta data: ${response}`);
      return response;
    })
    .catch((err) => {
      console.error('Retrieve meta data failed.', err);
      return '';
    });
}

Is there something wrong? thanks.

update

I can get project-id from metadata server correctly. Here is the code:

const METADATA_PROJECT_ID_URL = 'http://metadata.google.internal/computeMetadata/v1/project/project-id';

async function getProjectId() {
  const options = {
    headers: {
      'Metadata-Flavor': 'Google'
    }
  };

  return request(METADATA_PROJECT_ID_URL, options)
    .then((response) => {
      console.log('response: ', response);
      return response;
    })
    .catch((err) => {
      if (err && err.statusCode !== 200) {
        console.log('Error while talking to metadata server.');
        return 'Unknown_Project_ID';
      }
      return Promise.reject(err);
    });
}

which GAE environment? – Dan Cornilescu Nov 21 '18 at 10:53 — Dan Cornilescu, Nov 21 '18 at 10:53
@DanCornilescu standard environment – Lin Du Nov 21 '18 at 11:01 — Lin Du, Nov 21 '18 at 11:01

score 1 · Answer 1 · answered Nov 21 '18 at 12:06

A while ago this wasn't at all possible in the standard environment, see Is there a way to access the Google Cloud metadata service from AppEngine Standard for runtime configuration?

But things appear to be changing.

There is a mentioning of the Metadata service in the (1st generation) standard environment documentation, but:

only for the java sandbox
potentially limited scope - only a subset of the endpoints mentioned, maybe the user-configured aren't, indeed, covered. But may be a matter of interpretation (emphasis mine):

The following table lists the endpoints where you can make HTTP requests for specific metadata.

read-only:

Note: Metadata access is currently read only: you cannot write your own metadata for an instance.

This means the DNS limitation making it impossible a while ago was eliminated. Since you can get the data in the flexible environment it means it exists and you're not really trying to write it, so what you experience isn't related to the read-only limitation either.

It seems that indeed the service feature/endpoint you seek is more likely not available/functional, at least for the go sandbox (if not for all of them), rather than just an accidental documentation omission (which one might suspect/hope).

But the `metadata` can be retrieved in `GAE` flexible environment. I am pretty sure of this. What a `GCP`, waste my a lot of times. — Lin Du, Nov 21 '18 at 12:15
Yes, it's actually part of the recommended alternative for the [missing App identity](https://cloud.google.com/appengine/docs/flexible/python/migrating#app_identity). But that's a different environment. And I'm not certain the custom endpoints work in that case either. — Dan Cornilescu, Nov 21 '18 at 17:35

Lin Du · Accepted Answer · 2018-11-28T02:02:06.327

Finally I found the reason, it's metadata api version issue.

Instead of using

http://metadata.google.internal/computeMetadata/v1beta/project/attributes/${attr}

use

http://metadata.google.internal/computeMetadata/v1/project/attributes/${attr}

Now, I can get metadata from app engine flexible environment.

{"IT_EBOOKS_API":"http://it-ebooks-api.info/v1","PROJECT_ID":"just-aloe-212502","API_KEY":"12j28flsrbapznq"}

But for GAE standard environment and GCF. Still get 404 page not found

So I think but not sure that GCF and GAE standard environment are not running in compute engine.

GAE flexiable environment use compute engine as its infrastructure. That's why it can get metadata from compute engine.

score 0 · Answer 3 · answered Dec 21 '19 at 20:37

I had a similar error using python APIs from a compute engine VM. Refreshing the key file and using the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to it explicitly seemed to work. Even when that account was the only account configured with gcloud auth, I still had the error. I'm probably missing something in my understanding of where the Python libraries get their credentials from, but at least I have a workaround for the moment.

Get compute metadata from app engine, got 404 page not found error

3 Answers3