How to set environment variables using Google Cloud Build or other method in Google App Engine Standard Environment?

Question

Is there anyway to inject environment variables from Cloud Build into the App Engine Standard environment?

I do not want to push my environment variables to GitHub inside the app.yaml or .env. Thus, when Cloud Build pulls and deploys it is missing the .env file and the server is unable to complete some requests.

I am trying to avoid using Datastore as the async nature of Datastore will make the code a lot more messy. I tried to use encrypted secrets found here, but that doesn't seem to work as I added the secrets to app deploy and they do not make their way into the deployment, so I assume this is not the use case for Cloud Build.

I also tried the tutorial here, to import the .env file into App Engine Standard from storage, but since Standard does not have local storage I assume it goes into the void.

So is there anyway to inject the .env into App Engine Standard environment without using Datastore, or committing app.yaml or .env to change control? Potentially using Cloud Build, KMS, or some type of storage?

Here is what I tried for cloudbuild.yaml:

steps:
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
  secretEnv: ['SECRET1', 'SECRET2', 'SECRET3', 'SECRET4', 'SECRET5']
timeout: "1600s"

secrets:
- kmsKeyName: projects/<Project-Name>/locations/global/keyRings/<Key-Ring-Name>/cryptoKeys/<Key-Name>
  secretEnv:
    SECRET1: <encrypted-key-base64 here>
    SECRET2: <encrypted-key-base64 here>
    SECRET3: <encrypted-key-base64 here> 
    SECRET4: <encrypted-key-base64 here> 
    SECRET5: <encrypted-key-base64 here>

highfivebrian · Accepted Answer · 2019-06-29T06:18:30.080

29

Here is a tutorial on how to securely store env vars in your cloud build (triggers) settings and import them into your app.

Basically there are three steps:

Add your env vars to the 'variables' section in one of your build trigger settings

Screenshot of where to add variables in build triggers

By convention variables set in the build trigger must begin with an underscore (_)
Configure cloudbuild.yaml (on the second step in the code example) to read in variables from your build trigger, set them as env vars, and write all env vars in a local .env file

Add couldbuild.yaml (below) to your project root directory

steps:
- name: node:10.15.1
  entrypoint: npm
  args: ["install"]
- name: node:10.15.1
  entrypoint: npm
  args: ["run", "create-env"]
  env:
    - 'MY_SECRET_KEY=${_MY_SECRET_KEY}'
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
timeout: "1600s"

Add create-env script to package.json

"scripts": {
  "create-env": "printenv > .env"
},

Read env vars from .env to your app (config.js)

Install dotenv package

npm i dotenv -S

Add a config.js to your app

// Import all env vars from .env file
require('dotenv').config()

export const MY_SECRET_KEY = process.env.MY_SECRET_KEY

console.log(MY_SECRET_KEY) // => Hello

Done! Now you may deploy your app by triggering the cloud build and your app will have access to the env vars.

edited Jun 29 '19 at 06:18

answered Jun 29 '19 at 06:10

highfivebrian

457
5
7

Worked fantastic! Thank you! Also your article was great to follow along with. You're a real hero. – Alex Supkay Jun 30 '19 at 19:20
@AlexSupkay Glad you liked the solution. Thank you very much!! – highfivebrian Jun 30 '19 at 23:58
I am trying this solution but the substitute var in the .env file in gcloud is empty. Any idea ? – Christophe Chenel Feb 04 '21 at 00:09
I had problems with this solution but it was because I did not understand the steps well, in step 2 I did not understand that I had to manually edit the package.json file and add the sentence "" create-env ":" printenv> .env "" after doing this correctly, it worked perfect. – JORGE IVAN GUEVARA PEREZ Aug 09 '21 at 16:11
1

Looks like this solution doesn't work anymore. Had issues as other commenters on Medium. Ended up using CloudBuild trigger variables and separate app.yaml files – syldman Sep 23 '21 at 21:11
Great. Thanks you! – Alexandr Kazakov Oct 02 '21 at 22:25
Can you bake the .env file into the image? – Hakim Jan 02 '23 at 20:03
i didn't need the config.js step. otherwise worked great! – smoore4 Mar 03 '23 at 18:35
No need to do all the stuff involving npm - you can just make your build step say: - name: "bash" \n script: "echo MY_SECRET_KEY=${_MY_SECRET_KEY} > .env" – Matt Fletcher Mar 06 '23 at 11:17
Hate how stack overflow doesn't allow newlines... you get what i've done though hopefully – Matt Fletcher Mar 06 '23 at 11:17

score 3 · Answer 2 · answered Feb 06 '20 at 09:15

I have another solution, if someone is still interested in this. This should work on all languages, because environment variables are added directly into app.yaml file

Add substitution variable in build trigger (as described in this answer).
Add environment variables to app.yaml in a way they can be easily substituted with build trigger variables. Like this:

    env_variables:
     SECRET_KEY: %SECRET_KEY%

Add a step in cloudbuild.yaml to substitute all %XXX% variables inside app.yaml with their values from build trigger.

    - name: 'gcr.io/cloud-builders/gcloud'
      entrypoint: bash
      args:
      - '-c'
      - |
      sed -i 's/%SECRET_KEY%/'${_SECRET_KEY}'/g' app.yaml
      gcloud app deploy  app.yaml

weird, I am getting a no such file while running this sed -i 's/%SECRET_KEY%/'${_SECRET_KEY}'/g' app.yaml — Christophe Chenel, Feb 04 '21 at 00:32
The point of injecting these variables during build is having the secrets in a location with more restricted access than the code base. app.yaml is (and should be; as it serves other purposes than isolating the secrets) within the codebase. — mcyalcin, Jan 04 '23 at 12:10

treecoder · Answer 3 · 2023-03-06T19:24:46.007

You should NOT ^{NOT AT ALL ^please!} store secrets within any configuration store that keeps values in plain text. These types of security blunders are not acceptable in 2023, even for toy projects.

Config stores that keep plain text values are intended for storing things that are NOT secrets but values that are used in many places and are better off not repeated.

There is a reason, therefore, these stores have a word "variable" in their name, and not "secret".

Cloud Build

Github

All the build systems (should) come with two types of stores - one that keeps plain text values and other that stores encrypted secrets.

Cloud Build has secrets manager and Github has secrets. Use them for storing your tokens, keys, etc.

Check this out for Cloud Build - https://timothyreynolds.co.uk/words/2019/securing-environment-variables-in-google-cloud-build

Once you get your secrets from secrets manager, then you can do what other answers here are doing. But the source of the secrets should not be the plaintext config store.

this is also an excellent resource using the same approach: https://cloud.google.com/build/docs/securing-builds/use-secrets — smoore4, Mar 09 '23 at 22:11

score 0 · Answer 4 · edited Jun 20 '20 at 09:12

Based on your preferences that you have highlighted (Cloud Build, KMS). The Google Secrets link that you had mentioned involves storing sensitive data at build or runtime using Cloud KMS: KeyRing and CryptoKey. However, Google offers other Secret Management Solutions using Cloud KMS as well.

Here are a couple of other options you can use while storing Secrets:

Option 1 : You can store Secrets in code that are encrypted with a key from Cloud KMS. (This is typically used by encrypting your secret at the application layer.)

Benefit: Provides a layer of security from insider threats because it restricts access to the code with a corresponding key.

[You can find some additional information about these options on the Google Documentation here.]

Option 2: You can Store Secrets inside a Google Storage Bucket where your data is at rest encryption. (Similar to option 1 this has the ability to limit access to secrets to a small group of Developers.)

Benefit: Storing your secrets in a separate location ensures that if a breach of your code repository has occurred, your secrets may still be protected.)

[Note: Google recommends that you use two projects for proper separation of duties. One project will use Cloud KMS to manage the keys and the other project will use Cloud Storage to store the secrets.]

If the options listed above still do not meet your needs, I have found a StackOverflow question that shares a similar objective as your project. (i.e: Storing environment variables in GAE without Datastore)

The solution provided on this link illustrates the use of storing keys in a client_secrets.json file that gets excluded when uploading to git by listing it in .gitignore. You can find some Google examples (Python) of usage here.

I feel like this is a generic copy-pasta type answer that doesn't address the specific question as it applies to multiple-environment settings, using Node — Josh Noe, Mar 09 '19 at 23:24

score 0 · Answer 5 · answered Oct 02 '21 at 23:06

The highfivebrian answer is great, but I'm adding my slightly different solution.

1). In the root project folder we need the cloudbuild.yaml file but I'll call it buildsetttings.yaml, because first one name have a problem

In buildsetttings.yaml I added this code:

    steps:
  - name: node
    entrypoint: npm
    args: ['install']
  - name: node
    entrypoint: npm
    env:
      - 'DB_URL=${_DB_URL}'
      - 'SENDGRID_API_KEY=${_SENDGRID_API_KEY}'
      - 'CLIENT_ID=${_CLIENT_ID}'
    args: ['run', 'create-app-yaml']
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['app', 'deploy']

buildsetttings.yaml will be create app.yaml file in the Cloud Build, using a npm create-app-yaml command. Tip: app.yaml file we will then use to deploy our app to GCP App Engine.

2). In the root folder(near buildsetttings.yaml) we need to create create-app-yaml.js which will run in Cloud Build after it is called from buildsetttings.yaml.

In buildsetttings.yaml I added this code:

require('dotenv').config();

const fs = require('fs');

const appYamlContent = `runtime: nodejs14
env_variables:
  DB_URL: ${process.env.DB_URL}
  SENDGRID_API_KEY: ${process.env.SENDGRID_API_KEY}
  CLIENT_ID: ${process.env.CLIENT_ID}`;

fs.writeFileSync('./app.yaml', appYamlContent);

This code using a npm package dotenv(add it to package.json) and get variables from Cloud Build Trigger Variables and create with they app.yaml file.

3). app.yaml file was created in the Cloud build and our last step(name: 'gcr.io/cloud-builders/gcloud') in the buildsetttings.yaml, using app.yaml file, deploy the project to the Google Cloud App Engine.

Success!

In short, it works like this: buildsetttings.yaml run "create-app-yaml.js" in the Cloud Build, after which dynamically creates an app.yaml file by adding variables from Cloud Build Trigger Variables, then makes a deployment in the App Engine.

Notes:

Delete the file app.yamlin from you project, because it will be create dynamically in the Cloud Build. Also delete cloudbuild.yaml file, because instead we use buildsetttings.yaml.
package.json:

Cloud Build Trigger Variables:

score 0 · Answer 6 · answered Nov 13 '22 at 08:19

As of 2020/11/13. It seem like .env will work only at that step and in the next step an invisible .env will no longer there.

If you get stuck do try consume that printed .env it in 1 step like this ...
in cloudbuild.yaml

# [START cloudbuild_yarn_node]
steps:
  # Install
  - name: node
    entrypoint: yarn
    args: ["install"]
  # Build
  - name: node
    entrypoint: yarn
    env:
      - "FOO=${_FOO}"
    args: ["env-build"]

and in package.json add this

{
  "scripts": {
    "env-build": "printenv > .env && yarn build",
  }
}

in index.js

require('dotenv').config();
console.log(process.env.FOO);

Took me an hour to figure this out.

score 0 · Answer 7 · answered Jan 05 '23 at 10:54

First, I created secret using gcp secret manager and uploaded my env file there. Second, I called the secret in cloudbuild.yaml on run time and created a file with name of '.env' using echo.

Example

steps: 
- id: "Injecting ENV"
  name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: bash
  args:
      - '-c'
      - |
        echo $$ENV > .env
  secretEnv: ['ENV']


availableSecrets: 
  - versionName: projects/<Project-Name>/secrets/environment-variables/versions/1
    env: 'ENV'
timeout: 900s

How to set environment variables using Google Cloud Build or other method in Google App Engine Standard Environment?

7 Answers7

Linked