Istio adds and removed headers, but doesn't overwrite

Question

I am trying to add, overwrite and remove headers with VirtualServices, with Istio. Adding a header to the request and removing a header from the response works just fine, but it is not overwriting the header from the request.

So, according to Istio docs, headers operations are as follows:

And this is my VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-vs
spec:
  hosts:
  - server.istio.svc.cluster.local
  http:
  - headers:
      request:
        set:
          test: "true"
        add:
          added: "header"
    route:
    - destination:
        host: server.istio.svc.cluster.local
        subset: apache
      weight: 90
    - destination:
        host: server.istio.svc.cluster.local
        subset: nginx
      headers:
        response:
          remove:
          - foo
      weight: 10

Focusing on set a header, this is my understanding. Istio is going to identify a header with key test, and it is going to overwrite the value by true. So, if I curl my service with -H "test: hello" parameter, I should see test: true in my request.

Now, I have tcpdump installed in my backend pod, and I can see added: header. I can also see that foo: bar has been removed (my nginx server is returning foo: bar along with the response), but I can't see test: true header. Is my understanding correct about header operations with Istio?

From the tests:

REQUEST

/ # curl -H "test: hello" server
<h1>THIS IS AN ISTIO DEMO</h1>/ #

RESPONSE

15:47:27.387891 IP 192.168.49.58.39652 > 192.168.113.134.80: Flags [S], seq 2810811797, win 28000, options [mss 1400,sackOK,TS val 1900372734 ecr 0,nop,wscale 7], length 0
E..<d.@.>.....1:..q....P..........m`
3.....x...
qEb.........
15:47:27.388130 IP 192.168.49.58.39652 > 192.168.113.134.80: Flags [.], ack 2093848160, win 219, options [nop,nop,TS val 1900372735 ecr 3950379544], length 0
E..4d.@.>.....1:..q....P....|..`.....z.....
qEb..v..
15:47:27.388202 IP 192.168.49.58.39652 > 192.168.113.134.80: Flags [P.], seq 2810811798:2810812507, ack 2093848160, win 219, options [nop,nop,TS val 1900372735 ecr 3950379544], length 709: HTTP: GET / HTTP/1.1
E...d.@.>.....1:..q....P....|..`....S&.....
qEb..v..GET / HTTP/1.1
host: server
user-agent: curl/7.59.0
accept: */*
test: hello                  <-HERE IS THE HEADER THAT SUPPOSED TO BE OVERWRITTEN
x-forwarded-proto: http
x-request-id: 05ff2bb6-8810-9411-8a3b-7456066e1d16
x-envoy-decorator-operation: server-vs:80/*
x-istio-attributes: CjwKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBIgEh5zZXJ2ZXIuaXN0aW8uc3ZjLmNsdXN0ZXIubG9jYWwKOgoXZGVzdGluYXRpb24uc2VydmljZS51aWQSHxIdaXN0aW86Ly9pc3Rpby9zZXJ2aWNlcy9zZXJ2ZXIKJAoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEggSBnNlcnZlcgooCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIHEgVpc3Rpbwo6Cgpzb3VyY2UudWlkEiwSKmt1YmVybmV0ZXM6Ly9jdXJsZXItNzU1Y2M3Y2ZmZi14dmxiMi5pc3Rpbw==
x-b3-traceid: 741c90fab1ca5e23802399916451563e
x-b3-spanid: 802399916451563e
x-b3-sampled: 1
added: header                <- HERE IS THE ADDED HEADER
content-length: 0


15:47:27.390857 IP 192.168.49.58.39652 > 192.168.113.134.80: Flags [.], ack 2093848436, win 228, options [nop,nop,TS val 1900372737 ecr 3950379547], length 0
E..4d.@.>.....1:..q....P...[|..t...........
qEc..v..

But I was expecting to see test: true

Hi suren , I tested it and in fact i couldn't make it work with request field, but i made it with response one, if You change it, it should work. The question now is, if that's the answer You we're looking for or You want to make it work with request field? — Jakub, Dec 31 '19 at 10:15
Yeah, I needed it in request headers. I posted the answer that solved the problem. Weird thing. — suren, Jan 02 '20 at 10:20

score 5 · Accepted Answer · answered Jan 02 '20 at 10:19

A silly thing solved the problem. Looks like if I define first set then add, in the yaml file, it doesn't work, but if I do it backwords, it works. I guess it's a bug. So the following yaml worked:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-vs
spec:
  hosts:
  - server.istio.svc.cluster.local
  http:
  - headers:
      request:
        add:                    <- add goes first
          added: "header"
        set:                    <- set goes second
          test: "true"
    route:
    - destination:
        host: server.istio.svc.cluster.local
        subset: apache
      weight: 90
    - destination:
        host: server.istio.svc.cluster.local
        subset: nginx
      headers:
        response:
          remove:
          - foo
      weight: 10

Istio adds and removed headers, but doesn't overwrite

1 Answers1

Linked