istio-ingressgateway :Readiness probe failed: HTTP probe failed with statuscode: 503

Question

I am trying to setup istio1.5.1 in minicube kubernetes cluster,I'm following the official documentation of Knative for setting up istio without sidecar injection. I;m facing issue with istio ingress gateway service which is showing external ip of ingressgateway service as . I have gone through other answers posted here as well as many other forums but none of them helped in my case.

Using minikube v1.9.1 with driver=none helm v2.16.5 kubectl v1.18.0

I am getting the following output for: kubectl get pods --namespace istio-system

 NAME                                   READY   STATUS    RESTARTS   AGE
 istio-ingressgateway-b599cccd9-qnp5l   1/1     Running   0          60s
 istio-pilot-b67ccb85-mfllc             1/1     Running   0          60s

kubectl get svc --namespace istio-system

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                                      
AGE
istio-ingressgateway   LoadBalancer   10.104.37.189    ***<pending>***   15020:30168/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32576/TCP,15030:31080/TCP,15031:31767/TCP,15032:31812/TCP,15443:30660/TCP   74s
istio-pilot            ClusterIP      10.100.224.212   <none>     15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       74s

On describing ingress pod I am getting a warning Readiness probe failed: HTTP probe failed with statuscode: 503

Can someone help me to solve this issue. Thanks!

Updating with output from trying out the answer:

kubectl apply -f metallb.yaml

  podsecuritypolicy.policy/controller created
  podsecuritypolicy.policy/speaker created
  serviceaccount/controller created
  serviceaccount/speaker created
  clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
  clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
  role.rbac.authorization.k8s.io/config-watcher created
  role.rbac.authorization.k8s.io/pod-lister created
  clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
  clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
  rolebinding.rbac.authorization.k8s.io/config-watcher created
  rolebinding.rbac.authorization.k8s.io/pod-lister created
  daemonset.apps/speaker created
  deployment.apps/controller created

$ kubectl get pods -n metallb-system

  No resources found in metallb-system namespace.

After applying the yaml file it show that everything is created but I am not getting any pod deployed under metallb-system namespace.

Minikube doesn't give you an external IP (as it can't deploy a loadbalancer). Check out the answers to [this](https://stackoverflow.com/questions/55927602/how-to-expose-an-external-ip-address-for-a-sample-istio-application) and [this question](https://stackoverflow.com/questions/44110876/kubernetes-service-external-ip-pending). — char, Apr 07 '20 at 06:59
Thanks for your response! I have seen this mentioned in the documentation so I tried changing the port type from loadbalancer to nodeport it went successful but while describing the pod I am still getting the readiness probe warning. Does that fail my further steps? Can I start setting up Knative now? — Anjali A, Apr 07 '20 at 07:26

Harsh Manvar · Accepted Answer · 2020-04-07T13:32:31.363

Minikube may not provide external IP or loadbalancer you might have to use metalLB in minikube.

Metal lb : https://metallb.universe.tf/

you can also check this out for reference:https://medium.com/@emirmujic/istio-and-metallb-on-minikube-242281b1134b

This is also a good reference: https://gist.github.com/diegopacheco/9ed4fd9b9a0f341e94e0eb791169ecf9

Metal LB YAMl :

apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    app: metallb
---

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: metallb-system
  name: controller
  labels:
    app: metallb
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: metallb-system
  name: speaker
  labels:
    app: metallb

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metallb-system:controller
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
  resources: ["services/status"]
  verbs: ["update"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metallb-system:speaker
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["services", "endpoints", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: metallb-system
  name: config-watcher
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create"]
---

## Role bindings
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metallb-system:controller
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: controller
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metallb-system:speaker
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: speaker
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: metallb-system
  name: config-watcher
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: controller
- kind: ServiceAccount
  name: speaker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: config-watcher
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: metallb-system
  name: speaker
  labels:
    app: metallb
    component: speaker
spec:
  selector:
    matchLabels:
      app: metallb
      component: speaker
  template:
    metadata:
      labels:
        app: metallb
        component: speaker
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7472"
    spec:
      serviceAccountName: speaker
      terminationGracePeriodSeconds: 0
      hostNetwork: true
      containers:
      - name: speaker
        image: metallb/speaker:v0.7.1
        imagePullPolicy: IfNotPresent
        args:
        - --port=7472
        - --config=config
        env:
        - name: METALLB_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        ports:
        - name: monitoring
          containerPort: 7472
        resources:
          limits:
            cpu: 100m
            memory: 100Mi

        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - all
            add:
            - net_raw

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: metallb-system
  name: controller
  labels:
    app: metallb
    component: controller
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: metallb
      component: controller
  template:
    metadata:
      labels:
        app: metallb
        component: controller
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7472"
    spec:
      serviceAccountName: controller
      terminationGracePeriodSeconds: 0
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534 # nobody
      containers:
      - name: controller
        image: metallb/controller:v0.7.1
        imagePullPolicy: IfNotPresent
        args:
        - --port=7472
        - --config=config
        ports:
        - name: monitoring
          containerPort: 7472
        resources:
          limits:
            cpu: 100m
            memory: 100Mi

        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true

Could not see any resources under metallb-system namespace even after applying the manifests.. I mean, its not creating any service and controller pods under the namespace — Anjali A, Apr 07 '20 at 09:35
i am not 100% sure however if you have some time can try with KVM ? — Harsh Manvar, Apr 07 '20 at 13:10
Please see the result after using the metallb.yaml (correct: apiVersion for Deployment and Daemon set needs to be updated to apps/v1 for kube 1.9+) . I am not getting the pods deployed after using this also — Anjali A, Apr 07 '20 at 13:25
change API version in file where it's set for `apps/v1beta1` and `apps/v1beta2` to `apps/v1`. — Harsh Manvar, Apr 07 '20 at 13:30
Wasted so much time in installing VirtualBox, I think even the doc is not well maintained so I skipped it and set driver none. — Anjali A, Apr 07 '20 at 13:30
you are using higher thn 1.9 kubernetes version maybe 1.16 using currently. so apps/v1beta1 and apps/v1beta2 deprecated. — Harsh Manvar, Apr 07 '20 at 13:30
changin the api version is already done but thats not the problem — Anjali A, Apr 07 '20 at 13:33
please see the updated question. I have attached the output of metallb setup . Can you see its not showng any pods after applying the yml — Anjali A, Apr 07 '20 at 13:34
in your log there no line for namespaces created `metallb-system`. — Harsh Manvar, Apr 07 '20 at 13:43
`namespace/metallb-system created serviceaccount/controller created serviceaccount/speaker created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created role.rbac.authorization.k8s.io/config-watcher created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/config-watcher created daemonset.apps/speaker created deployment.apps/controller created` my log — Harsh Manvar, Apr 07 '20 at 13:44
namespace/metallb-system unchanged serviceaccount/controller unchanged serviceaccount/speaker unchanged ...... its already created — Anjali A, Apr 07 '20 at 18:42
But , kubectl get pods -n metallb-system -----> gives No resources found in metallb-system namespace. — Anjali A, Apr 07 '20 at 18:45

istio-ingressgateway :Readiness probe failed: HTTP probe failed with statuscode: 503

1 Answers1