Setting a trace id in nginx load balancer

Question

I'm using nginx as a load balancer in front of several upstream app servers and I want to set a trace id to use to correlate requests with the app server logs. What's the best way to do that in Nginx, is there a good 3rd party module for this?

Otherwise a pretty simple way would be to base it off of timestamp (possibly plus a random number if that's not precise enough) and set it as an extra header on the request, but the only set_header command I see in the docs is for setting a response header.

isapir · Answer 1 · 2016-10-13T18:20:10.560

108

nginx 1.11.0 added the new variable $request_id which is a unique identifier, so you can do something like:

   location / {
      proxy_pass http://upstream;
      proxy_set_header X-Request-Id $request_id;
   }

See reference at http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_id

edited Oct 13 '16 at 18:20

answered Jul 19 '16 at 00:34

isapir

21,295
13
115
116

3

For reference: documentation on $request_id is here http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_id – Andrew Newdigate Aug 12 '16 at 13:20
`$request_id` has already been deprecated and removed in current versions of nginx. In case somebody wonders nowadays. You can use `$connection` (connection serial number) now. – Gerald Jul 21 '18 at 14:08
@Gerald can you provide a reference? I don't see anything about the deprecation – isapir Jul 22 '18 at 19:22
1

@isapir It's the other way round (didn't check that, sorry). `$request_id` is only available from nginx Plus version 1.11: [https://docs.nginx.com/nginx/releases/#nginx-plus-release-10-r10](https://docs.nginx.com/nginx/releases/#nginx-plus-release-10-r10) - For others, use this combination: `$connection-$connection_requests` which creates a unique request identifier as well (keeps the same for internal sub-requests). – Gerald Jul 23 '18 at 22:38
@Gerald - that's the reason my answer opens with "nginx 1.11.0 added ..." ;) – isapir Jul 24 '18 at 01:58
@isapir - yeah, I guess :D Are we really supposed to read things on the Internet starting at the top? – Gerald Jul 27 '18 at 12:43
`proxy_set_header X-Request-Id $connection-$connection_requests-$msec;` is recommended. – Justin Oct 17 '18 at 02:48
How to add `X-Request-Id` only when it is not set in request? If request header already have a request id, I don't want to rest it to a new one. – Sisir May 08 '20 at 11:05

score 33 · Accepted Answer · answered Jul 23 '14 at 12:33

33

In most cases you don't need a custom module, you can simply set a header with a combination of embedded variables of http_core_module which is (most probably) unique. Example:

  location / {
      proxy_pass http://upstream;
      proxy_set_header X-Request-Id $pid-$msec-$remote_addr-$request_length;
  }

This would yield a request id like "31725-1406109429.299-127.0.0.1-1227" and should be "unique enough" to serve as a trace id.

answered Jul 23 '14 at 12:33

sbange

626
6
8

very nice. can it be set to a variable for reuse within the request? so that it can be used in logs as well? otherwise calling $msec twice will generate a different value, so I'm thinking something like `set $request_id $pid-$msec-$remote_addr-$request_length;` but so far couldn't get it to work – isapir Feb 04 '15 at 19:13
6

it is working for me, I'm using this `set $tid $pid-$msec-$remote_addr-$request_length-$connection;` and `proxy_set_header X-Tracing-Id $tid;` later on, check your scope of usage... – sbange Feb 19 '15 at 11:03
Probably don't want to leak the $remote_addr incase users want to share request id's publically. – Justin Oct 17 '18 at 02:46
If nginx could take md5 or sha from `$pid-$msec-$remote_addr-$request_length` it is almost random uid and value will be monolythic – gayavat Oct 21 '19 at 10:06

Gerald · Answer 3 · 2018-10-23T05:25:28.773

Old question, new answer suitable for nginx verions 1.3.8, 1.2.5 and above.

You can use a combination of $connection and $connection_requests now. Just define your own variable in the server block:

server {
    ...
    set  $trace_id  $connection-$connection_requests;
    ...
}

This id is going to be unique across nginx unless the server gets restarted.

$connection - The connection serial number. This is a unique number assigned by nginx to each connection. If multiple requests are received on a single connection, they will all have the same connection serial number. Serial numbers reset when the master nginx process is terminated, so they will not be unique over long periods of time.

$connection_requests - The number of requests made through this $connection.

Then, in your location block, set the actual trace ID:

location / {
    ...
    proxy_set_header X-Request-Id $trace_id;
    ...
}

Bonus: Make $trace_id unique even after server restarts:

set  $trace_id  $connection-$connection_requests-$msec;

$msec - The current unix timestamp in seconds, with millisecond resolution (float).

I am using a variable as TRACE_ID in our application. Will ```set TRACE_ID $connection-$connection_requests; works for me ?``` because proxy_set_header is not working for me with this custom variable. — mohit, Feb 04 '19 at 18:08

score 0 · Answer 4 · answered Jul 19 '13 at 16:04

In our production environment we do have a custom module like this. It can generate a unique trace id and then it will be pushed into http headers which send to the upstream server. The upstream server will check if the certain field is set, it will get the value and write it to access_log, thus, we can trace the request.

And I find an 3rd party module just looks the same: nginx-operationid, hope it is helpful.

Setting a trace id in nginx load balancer

4 Answers4

Linked