Istio: Health check / sidecar fails when I enable the JWT RequestAuthentication

Question

OBSOLETE:

I keep this post for further reference, but you can check better diagnose (not solved yet, but workarounded) in

Istio: RequestAuthentication jwksUri does not resolve internal services names

UPDATE:

In Istio log we see the next error. uaa is a kubernetes pod serving OAUTH authentication/authorization. It is accessed with the name uaa from the normal services. I do not know why the istiod cannot find uaa host name. Have I to use an specific name? (remember, standard services find uaa host perfectly)

2021-03-03T18:39:36.750311Z     error   model   Failed to fetch public key from "http://uaa:8090/uaa/token_keys": Get "http://uaa:8090/uaa/token_keys": dial tcp: lookup uaa on 10.96.0.10:53: no such host
2021-03-03T18:39:36.750364Z     error   Failed to fetch jwt public key from "http://uaa:8090/uaa/token_keys": Get "http://uaa:8090/uaa/token_keys": dial tcp: lookup uaa on 10.96.0.10:53: no such host
2021-03-03T18:39:36.753394Z     info    ads     LDS: PUSH for node:product-composite-5cbf8498c7-jd4n5.chp18 resources:29 size:134.3kB
2021-03-03T18:39:36.754623Z     info    ads     RDS: PUSH for node:product-composite-5cbf8498c7-jd4n5.chp18 resources:14 size:14.2kB
2021-03-03T18:39:36.790916Z     warn    ads     ADS:LDS: ACK ERROR sidecar~10.1.1.56~product-composite-5cbf8498c7-jd4n5.chp18~chp18.svc.cluster.local-10 Internal:Error adding/updating listener(s) virtualInbound: Provider 'origins-0' in jwt_authn config has invalid local jwks: Jwks RSA [n] or [e] field is missing or has a parse error

2021-03-03T18:39:55.618106Z     info    ads     ADS: "10.1.1.55:41162" sidecar~10.1.1.55~review-65b6886c89-bcv5f.chp18~chp18.svc.cluster.local-6 terminated rpc error: code = Canceled desc = context canceled

Original question

I have a service that is working fine, after injecting istio sidecar to a standard kubernetes pod.

I'm trying to add jwt Authentication, and for this, I'm following the official guide Authorization with JWT

My problem is

• If I create the JWT resources (RequestAuthorization and AuthorizationPolicy) AFTER injecting the istio dependencies, everything (seems) to work fine
• But if I create the JWT resources (RequestAuthorization and AuthorizationPolicy) and then inject the Istio the pod doesn't start. After checking the logs, seems that the sidecar is not able to work (maybe checking the health?)

My code:

1. JWT Resources

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "ra-product-composite"
spec:
  selector:
    matchLabels:
      app: "product-composite"
  jwtRules:
  - issuer: "http://uaa:8090/uaa/oauth/token"
    jwksUri: "http://uaa:8090/uaa/token_keys"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: "ap-product-composite"
spec:
  selector:
    matchLabels:
      app: "product-composite"
  action: ALLOW
  # rules:
  # - from:
  #   - source:
  #      requestPrincipals: ["http://uaa:8090/uaa/oauth/token/faf5e647-74ab-42cc-acdb-13cc9c573d5d"]
  # b99ccf71-50ed-4714-a7fc-e85ebae4a8bb

2- I use destination rules as follows

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dr-product-composite
spec:
  host: product-composite
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

3- My service deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-composite
spec:
  replicas: 1
  selector:
    matchLabels:
      app: product-composite
  template:
    metadata:
      labels:
        app: product-composite
        version: latest
    spec:
      containers:
      - name: comp
        image: bthinking/product-composite-service
        imagePullPolicy: Never
        env:
          - name: SPRING_PROFILES_ACTIVE
            value: "docker"
          - name: SPRING_CONFIG_LOCATION
            value: file:/config-repo/application.yml,file:/config-repo/product-composite.yml
        envFrom:
          - secretRef:
              name: rabbitmq-client-secrets            
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: 350Mi
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /actuator/info
            port: 4004
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 20
          successThreshold: 1
        readinessProbe:
          httpGet:
            scheme: HTTP
            path: /actuator/health
            port: 4004
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
          successThreshold: 1
        volumeMounts:
        - name: config-repo-volume
          mountPath: /config-repo
      volumes:
      - name: config-repo-volume
        configMap:
          name: config-repo-product-composite
---
apiVersion: v1
kind: Service
metadata:
  name: product-composite
spec:
  selector:
    app: "product-composite"
  ports:
    - port: 80
      name: http
      targetPort: 80
    - port: 4004
      name: http-mgm
      targetPort: 4004

4- Error log in the pod (combined service and sidecar)

2021-03-02 19:34:41.315 DEBUG 1 --- [undedElastic-12] o.s.s.w.s.a.AuthorizationWebFilter       : Authorization successful
2021-03-02 19:34:41.315 DEBUG 1 --- [undedElastic-12] .b.a.e.w.r.WebFluxEndpointHandlerMapping : [0e009bf1-133] Mapped to org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler@e13aa23
2021-03-02 19:34:41.316 DEBUG 1 --- [undedElastic-12] ebSessionServerSecurityContextRepository : No SecurityContext found in WebSession: 'org.springframework.web.server.session.InMemoryWebSessionStore$InMemoryWebSession@48e89a58'
2021-03-02 19:34:41.319 DEBUG 1 --- [undedElastic-15] .s.w.r.r.m.a.ResponseEntityResultHandler : [0e009bf1-133] Using 'application/vnd.spring-boot.actuator.v3+json' given [*/*] and supported [application/vnd.spring-boot.actuator.v3+json, application/vnd.spring-boot.actuator.v2+json, application/json]
2021-03-02 19:34:41.320 DEBUG 1 --- [undedElastic-15] .s.w.r.r.m.a.ResponseEntityResultHandler : [0e009bf1-133] 0..1 [java.util.Collections$UnmodifiableMap<?, ?>]
2021-03-02 19:34:41.321 DEBUG 1 --- [undedElastic-15] o.s.http.codec.json.Jackson2JsonEncoder  : [0e009bf1-133] Encoding [{}]
2021-03-02 19:34:41.326 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations     : [id: 0x0e009bf1, L:/127.0.0.1:4004 - R:/127.0.0.1:57138] Detected non persistent http connection, preparing to close
2021-03-02 19:34:41.327 DEBUG 1 --- [or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter    : [0e009bf1-133] Completed 200 OK
2021-03-02 19:34:41.327 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations     : [id: 0x0e009bf1, L:/127.0.0.1:4004 - R:/127.0.0.1:57138] Last HTTP response frame
2021-03-02 19:34:41.328 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations     : [id: 0x0e009bf1, L:/127.0.0.1:4004 - R:/127.0.0.1:57138] Last HTTP packet was sent, terminating the channel
2021-03-02T19:34:41.871551Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected

5- Istio injection

kubectl get deployment product-composite  -o yaml | istioctl kube-inject -f - | kubectl apply -f -

NOTICE: I have checked a lot of post in SO, and it seems that health checking create a lot of problems with sidecars and other configurations. I have checked the guide Health Checking of Istio Services with no success. Specifically, I tried to disable the sidecar.istio.io/rewriteAppHTTPProbers: "false", but it is worse (in this case, doesn't start neither the sidecar neither the service.