How to create kubernetes secret as json object and load the same in kubernetes environment as json

Question

I need to pass a JWK as kubernetes environment variable to my app.

I created a file to store my key like so:

cat deploy/keys/access-signature-public-jwk 
{
  algorithm = "RS256"
  jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
}

Which is then used to create a kubernetes secret like so:

kubectl create secret generic intimations-signature-public-secret --from-file=./deploy/keys/access-signature-public-jwk

Which is then retrived in the kubernetes environment variable as:

- name: ACCESS_SIGNATURE_PUBLIC_JWK
  valueFrom:
    secretKeyRef:
      name: intimations-signature-public-secret
      key: access-signature-public-jwk

And passed to the application.conf of the application likeso:

pac4j.lagom.jwt.authenticator {
  signatures = [
    ${ACCESS_SIGNATURE_PUBLIC_JWK}
  ]
}

The pac4j library expects the config pac4j.lagom.jwt.authenticator as a json object. But get the following exception when I run this app:

com.typesafe.config.ConfigException$WrongType: env variables: signatures has type list of STRING rather than list of OBJECT
    at com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:452)
    at com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:460)
    at com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:465)
    at org.pac4j.lagom.jwt.JwtAuthenticatorHelper.parse(JwtAuthenticatorHelper.java:84)
    at com.codingkapoor.holiday.impl.core.HolidayApplication.jwtClient$lzycompute(HolidayApplication.scala

POD Description

Name:         holiday-deployment-55b86f955d-9klk2
Namespace:    default
Priority:     0
Node:         minikube/192.168.99.103
Start Time:   Thu, 28 May 2020 12:42:50 +0530
Labels:       app=holiday
              pod-template-hash=55b86f955d
Annotations:  <none>
Status:       Running
IP:           172.17.0.5
IPs:
  IP:           172.17.0.5
Controlled By:  ReplicaSet/holiday-deployment-55b86f955d
Containers:
  holiday:
    Container ID:   docker://18443cfedc7fd39440f5fa6f038f36c58cec1660a2974e6432500e8c7d51f5e6
    Image:          codingkapoor/holiday-impl:latest
    Image ID:       docker://sha256:6e0ddcf41e0257755b7e865424671970091d555c4bad88b5d896708ded139eb7
    Port:           8558/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:49:24 +0530
      Finished:     Thu, 28 May 2020 22:49:29 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 May 2020 22:44:15 +0530
      Finished:     Thu, 28 May 2020 22:44:21 +0530
    Ready:          False
    Restart Count:  55
    Liveness:       http-get http://:management/alive delay=20s timeout=1s period=10s #success=1 #failure=10
    Readiness:      http-get http://:management/ready delay=20s timeout=1s period=10s #success=1 #failure=10
    Environment:
      JAVA_OPTS:                     -Xms256m -Xmx256m -Dconfig.resource=prod-application.conf
      APPLICATION_SECRET:            <set to the key 'secret' in secret 'intimations-application-secret'>  Optional: false
      MYSQL_URL:                     jdbc:mysql://mysql/intimations_holiday_schema
      MYSQL_USERNAME:                <set to the key 'username' in secret 'intimations-mysql-secret'>                                 Optional: false
      MYSQL_PASSWORD:                <set to the key 'password' in secret 'intimations-mysql-secret'>                                 Optional: false
      ACCESS_SIGNATURE_PUBLIC_JWK:   <set to the key 'access-signature-public-jwk' in secret 'intimations-signature-public-secret'>   Optional: false
      REFRESH_SIGNATURE_PUBLIC_JWK:  <set to the key 'refresh-signature-public-jwk' in secret 'intimations-signature-public-secret'>  Optional: false
      REQUIRED_CONTACT_POINT_NR:     1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kqmmv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-kqmmv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kqmmv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From               Message
  ----     ------   ----                   ----               -------
  Normal   Pulled   5m21s (x23 over 100m)  kubelet, minikube  Container image "codingkapoor/holiday-impl:latest" already present on machine
  Warning  BackOff  27s (x466 over 100m)   kubelet, minikube  Back-off restarting failed container

I was wondering if there is any way to pass the environment variable as a json object instead of string. Please suggest. TIA.

could you please share the output of `env` command of your pod? — hariK, May 28 '20 at 09:40
@hariK Pod failed to start: `holiday-deployment-55b86f955d-9klk2 0/1 CrashLoopBackOff 53 9h` Cause of which we can't print environments `kubectl exec holiday-deployment-55b86f955d-9klk2 -- printenv error: unable to upgrade connection: container not found ("holiday")` — iamsmkr, May 28 '20 at 17:14
@ShivamKapoor could you please share the output of `kubectl describe po`? — hariK, May 28 '20 at 17:19

hariK · Answer 1 · 2020-05-28T20:54:05.287

First, the file access-signature-public-jwk is not a valid JSON file. You should update it as a valid one.

{
  "algorithm" : "RS256",
  "jwk" : {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
}

Steps I followed to validate.

kubectl create secret generic token1 --from-file=jwk.json

Mount the secret into the pod.

env:
  - name: JWK
    valueFrom:
      secretKeyRef:
        name: token
        key: jwk.json

exec to the pod and check the env variable JWK

$ echo $JWK
{ "algorithm" : "RS256", "jwk" : {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"} }

Copy the content to a file

echo $JWK > jwk.json

Validate the file

$ jsonlint-php jwk.json 
Valid JSON (jwk.json)

If I use the file you are given and followed the same steps. It gives an json validation error. Also, env variables are always strings. You have to convert them into the required types in your code.

$ echo $JWK
{ algorithm = "RS256" jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"} }
$ echo $JWK > jwk.json
$ jsonlint-php jwk.json 
jwk.json: Parse error on line 1:
{ algorithm = "RS256" 
-^
Expected one of: 'STRING', '}'

Also, I don't see any difference in steps I took to create the secret and use it as an environment than what you mentioned in your answer. Probably, I failed to draw the attention to the issue here, which is that when the environment gets read in the pod, it gets read as string and not json causing the trouble. `echo $JWK > jwk.json` is always going to resolve into a valid json anyway. — iamsmkr, May 28 '20 at 20:12
I too was heading to the similar conclusion. Thanks for validating the same. Also, besides passing secret as env var and converting the same to json in code, is there any other alternative way to save and read the secret as json? I was wondering if mounting secrets as data volumes, as suggested here https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod, would help! — iamsmkr, May 28 '20 at 21:05

score 2 · Answer 2 · answered May 29 '20 at 19:05

Although not a direct answer but an alternate solution to this problem.

As @hariK pointed out environment variables are always strings and in order to consume them as json we would need to convert the env var read as string into json.

However, in my case, this was not a viable solution because I was using a lib that was expecting a Config object and not a json object directly which would have meant a lot of work. Converting string -> json -> Config. Plus this approach is inconsistent with how Config object was being built in the developement scenarios i.e., json -> Config. See here.

The framework I am using to build this app is based on Play Framework which allows to modularize application configs in separate files and then club the required pieces together in a desired config file, as shown below. You can read it more in detail here.

application.conf

include "/opt/conf/app1.conf"
include "/opt/conf/app2.conf"

This allowed me to make use of Using Secrets as files from a Pod feature from kubernetes.

Basically, I created a small config file that contains a part of my main application configuration file, as shown below:

cat deploy/keys/signature-public-jwk 
pac4j.lagom.jwt.authenticator {
  signatures = [
    {
      algorithm = "RS256"
      jwk = {"kty":"RSA","e":"AQAB","n":"ghhDZxuUo6TaSvAlD23mLP6n_T9pQuJsFY4JWdBYTjtcp_8Q3QeR477jou4cScPGczWw2JMGnx-Ao_b7ewagSl7VHpECBFHgcnlAgs5j6jfnd3M9ADKD2Yc756iXlIMT9xKDblIcXQQYlXalqxGvnLRLv1KAgVVVpVWzQd6Iz8WdTnexVrh7L9N87QQbOWcAVWGHCWCLCBsVE7JbC-XDt9h9P1g1sMqMV-qp7HjSXUKWuF2NwOnL2VeFSED7gdefs2Za1UYqhfwxdGl7aaPDXhjib0cfg4NvbcXMzxDEVkeJqhdDfD82wHOs4qFvnFMVxq9n6VVExSxsJq8gBJ7Z2AmfoXpmZC1L1ZwULB2KKpFXDCzgBELPLrfyIf8mNnk2nuuLT-aaMsqy2uB-ea3du4lyWo9MLk6x-L5g-n1oADKFKBY9aP2QQwruCG92XSd7jA9yLtbgr9OGVCYezxIxFp4vW6KcmPwJQjozWtwkZjeo4hv-zhRac73WDox2hDkif7WPTuEvC21fRy3GvyPIUPKPJA8pJjb2TXT7DXknR97CTnOWicuh3HMoRlVIwUzM5SVLGSXex0VjHZKgLYwQYukg5O2rab_4NxpD6LqLHx1bbPssC7BedCIfWX1Vcae40tlfvJAM09MiwQPZjWRahW_fK_9X5F5_rtUhCznm32M"}
    }
  ]
}

Then created a kubernetes secret and mounted volumes in deployment to appear in the pod as file

kubectl create secret generic signature-public-secret --from-file=./deploy/secrets/signature-public-jwks.conf

// deployment yaml
    spec:
      containers:
        - name: employee
          image: "codingkapoor/employee-impl:latest"
          volumeMounts:
            - name: signature-public-secret-conf
              mountPath: /opt/conf/signature-public-jwks.conf
              subPath: signature-public-jwks.conf
              readOnly: true
      volumes:
        - name: signature-public-secret-conf
          secret:
            secretName: signature-public-secret

Use this mounted file location in the application.conf to include the same

include file("/opt/conf/signature-public-jwks.conf")

Notice that the mountPath and the file location in the application.conf are same.

Advantages of this approach:

The solution is consistent with both the development and test, production environments as we could return json instead of string to the lib, as explained above
Secrets shouldn't be passed as environment variables anyway! You can read more about it here.

How to create kubernetes secret as json object and load the same in kubernetes environment as json

2 Answers2