Why these ingress rules don't expose a service to outside? (NET::ERR_CERT_AUTHORITY_INVALID/ERR_TOO_MANY_REDIRECTS)

Question

I'm trying to setup and expose a service (ArgoCD) to outside a cluster. Note: I'm fairly new to Kubernetes, so quite probably I have some misconceptions. If you can see one, please help me get rid of it. If more information is needed to diagnose what's happening, please let me know, I'll add it.

I have nginx-ingress ingress controller installed in the cluster in the namespace nginx. I have installed ArgoCD via helm into argocd namespace*. kubectl get service -n argocd shows (omitting AGE column):

NAME                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
projectname-argocd-application-controller   ClusterIP   10.100.249.133   <none>        8082/TCP
projectname-argocd-dex-server               ClusterIP   10.100.80.187    <none>        5556/TCP,5557/TCP
projectname-argocd-redis                    ClusterIP   10.100.230.170   <none>        6379/TCP
projectname-argocd-repo-server              ClusterIP   10.100.221.87    <none>        8081/TCP
projectname-argocd-server                   ClusterIP   10.100.22.26     <none>        80/TCP,443/TCP

As far as I understand, service projectname-argocd-server is the one I should expose to get ArgoCD WebUI. Trying to do so, I've created an ingress (based on docs):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-routing

spec:
  rules:
  - host: test2.projectname.org
    http:
      paths:
      - path: /
        pathType: Prefix # presumably may comment this out
        backend:
          service:
            name: projectname-argocd-server
            port:
              number: 80
  # this was added later while trying to figure the problem out
  defaultBackend:
    service:
      name: projectname-argocd-server
      port:
        number: 80
  ingressClassName: nginx

and applied it via kubectl apply -f routing.yaml -n argocd. Now I can see the ingress is created along with the one created by deployment of ArgoCD, and the output of kubectl get ing -A is (omitting AGE, and PORTS that are 80; <url> is url of LoadBalancer shown in AWS console):

NAMESPACE   NAME                        CLASS    HOSTS                   ADDRESS
argocd      projectname-argocd-server   nginx    test.projectname.org    <url>
argocd      ingress-routing             nginx    test2.projectname.org   <url>

By the way, kubectl get svc -n nginx shows that nginx-ingress-ingress-nginx-controller is LoadBalancer with url <url> (80:30538/TCP).

kubectl describe ingress -n argocd shows that ingress ingress-routing is ok, with correct address, default backend and rules; for ingress projectname-argocd-server it shows ok address and rules (path /), although Default backend is shown as default-http-backend:80 (<error: endpoints "default-http-backend" not found>).

Now let me also show the DNS settings to complete the picture:

• I've created a hosted zone for projectname.org (in Route 53), put its DNS servers to NS-entries of domain register
• I've created a CNAME entry in the hosted zone, pointing test.projectname.org to <url>
• I've created an A entry for test2.projectname.org, selected the load balancer from the list and so it points to dualstack.<url>

I expected to see ArgoCD interface at least at one of http://test.projectname.org/ and http://test2.projectname.org/. What actually happens is:

1. when I open http://test.projectname.org/, it redirects me to https url and shows NET::ERR_CERT_AUTHORITY_INVALID. If I insist on visiting, browser shows ERR_TOO_MANY_REDIRECTS.

2. Before I added ingress class and moved ingress-routing from nginx namespace to argocd namespace, http://test2.projectname.org/ gave me 404; now it also redirects to https and then gives ERR_TOO_MANY_REDIRECTS

3. I've also checked the /healthz addresses but they give the same result as the / ones. (in contrast, http://<url>/healthz gives an empty page)

My question is: what else am I missing, why I don't get the UI?

Is it impossible to expose a service before setting some SSL certificate? Can 2 ingresses conflict when trying to expose the same thing on different subdomains (test.projectname.org and test2.projectname.org)? Can I see at least one service (ArgoCD) without using projectname.org to check if it is configured and deployed properly? (to separate if it's an ingress/routing/dns issue or a configuration issue)

(*) Here's the chart that I used to install ArgoCD:

apiVersion: v2
name: argo-cd
appVersion: v2.1.5
description: A declarative, GitOps continuous delivery tool for Kubernetes
version: 3.26.3

dependencies:
  - name: argo-cd
    version: 3.26.3
    repository: https://argoproj.github.io/argo-helm

and values-overwrite.yaml that I've used is just default values wrapped into argo-cd: thing since these should be applied to the dependency. Notably, those have enabled: false in ingress:, so the fact that ingress projectname-argocd-server is created is somewhat unexpected.

PS the nginx IngressClass was generated, not created manually, so it may be useful to see it as well (I've substituted ids and timestamps with "..."), as shown by kubectl get IngressClass nginx -o yaml:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: nginx
  creationTimestamp: ...
  generation: 1
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 1.0.3
    helm.sh/chart: ingress-nginx-4.0.5
  name: nginx
  resourceVersion: "5750"
  uid: ...
spec:
  controller: k8s.io/ingress-nginx

Answer 1

Right, the issue was somewhat complicated, but I've figured it out. Basically, it consists of 2 problems:

1. https configuration and
2. ingress configuration

The main problem about https configuration was solved in a separate question and is reduced to switching ACME server from staging to production. I've provided more details it in my answer.

Now, the ingress configuration is somewhat tricky since ArgoCD has some redirections, ~inner TLS requirements~, and also serves more than one protocol at :443. Fortunately, I've found this tutorial which shows ssl-passthrough settings, more ingress annotations, including nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" which fixes the ERR_TOO_MANY_REDIRECTS error. Here's my ingress config which works fine with https set up (note also changes in port and tls secret):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-argocd-routing
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: <cluster issuer name>
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - test2.projectname.org # switched to argocd. later
      secretName: argocd-secret # do not change, this is provided by Argo CD
  rules:
    - host: test2.projectname.org # switched to argocd. later
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: projectname-argocd-server
                port:
                  number: 443