Github Actions for dbt: parse secret

Question

I want to setup Github Actions for a dbt project and I need to parse the content of a json key from GCP to the profile.yml at the root of my project. I have set-up a repository secret in Github and just pasted the content of the json file in it. My workflow.yml looks like this:

name: Build and Push Docker Image to Google Cloud Artifact Registry

on:
  push:
    branches:
      - feature_branch1

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
    - name: Check Out Repository
      uses: actions/checkout@v3
      
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Build Image
      uses: docker/build-push-action@v2
      with:
        context: .
        push: false
        load: true
        tags: bi-test-image:latest
        build-args: |
          SERVICE_KEY=${{ secrets.SERVICE_KEY }}

    - name: Run dbt tests
      run: |
        docker run -e SERVICE_KEY=$SERVICE_KEY bi-test-image:latest /bin/bash -c "dbt deps && dbt source snapshot-freshness"

    - name: Login to Google Cloud Registry
      uses: docker/login-action@v1
      with:
        registry: europe-west1-docker.pkg.dev
        username: _json_key
        password: ${{ secrets.BI_ARTIFACTORY_SECRET }}

    - name: Configure Docker to use the gcloud command-line tool as a credential helper
      run: gcloud auth configure-docker europe-west1-docker.pkg.dev-docker.pkg.dev

    - name: Push Image to Google's Artifact Registry
      if: ${{ success() }}
      uses: docker/build-push-action@v2
      with:
        context: .
        push: true
        tags: europe-west1-docker.pkg.dev/project-id/bi-artifacts/bi-test-image:latest

and this is how my Dockerfile looks:

FROM python:3.11-slim-buster


WORKDIR /app

COPY dbt .

ARG SERVICE_KEY
        
# Write the service key to a file
RUN echo "${SERVICE_KEY}" > /service_account_key.json

# Install dbt
RUN pip install --no-cache-dir dbt-bigquery

# Run dbt when the container launches
RUN dbt deps && dbt source snapshot-freshness

My profiles.yml looks like this:

expresssteuer_data_platform:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account-json
      service_account_json: "{{ env_var('SERVICE_KEY') }}"
      project: "project-id"
      dataset: "dwhdev"
      location: "europe-west1"
      threads: 8
      timeout_seconds: 600
      priority: interactive
      keyfile: '/service_account_key.json'

dbt source snapshot-freshness throws an error:

Database Error
'NoneType' object has no attribute 'keys'

Any idea what I am doing wrong?

Thanks a lot in advance

Does this answer your question? [Get environment variable value in Dockerfile](https://stackoverflow.com/questions/19537645/get-environment-variable-value-in-dockerfile) — Azeem, Jun 20 '23 at 16:51
Hi Azeem, thanks for your reply, it brought me one step further indeed. Now I am facing another issue. I have updated my post accordingly — Moritz, Jun 20 '23 at 18:30
Hi Mortiz, you're welcome! See this already reported issue (https://github.com/dbt-labs/dbt-core/issues/2755). — Azeem, Jun 21 '23 at 04:34

score 1 · Accepted Answer · answered Jun 22 '23 at 07:54

I could solve it, I needed to store the content of the service account key base64 decoded. This is the part of my workflow where I use it:

- name: Build Image and run tests
  uses: docker/build-push-action@v2
  env:
    SERVICE_KEY64: ${{ secrets.SERVICE_KEY64 }}
  with:
    credentials_json: $SERVICE_KEY64
    context: .
    push: true
    tags: europe-west1-docker.pkg.dev/project-id/bi-artifacts/bi-test-image:latest
    build-args: |
      SERVICE_KEY64

This is my profiles.yml:

  bi_data_platform:
    target: dev
    outputs:
      dev:
        type: bigquery
        method: service-account
        keyfile: "/project-id/dbt/dbt-service-account.json"
        project: "project-id"
        dataset: "dwhdev"
        location: "europe-west1"
        threads: 8
        timeout_seconds: 600
        priority: interactive

And my Dockerfile:

FROM python:3.11-slim-buster

WORKDIR /repo-name

COPY . .

ARG SERVICE_KEY64

RUN apt-get update && apt-get install -y git && \
    echo ${SERVICE_KEY64} | base64 -d > dbt/dbt-service-account.json && \
    cat dbt/dbt-service-account.json &&  \
    cd dbt && \
    ls -l && \
    pip install --no-cache-dir dbt-bigquery && \
    dbt deps && dbt debug

Github Actions for dbt: parse secret

1 Answers1