Why RestTemplate hangs when called concurrently?

Question

I have a Spring Boot based application and using RestTemplate for sending HTTP requests. Some endpoints to which the application sends requests respond very slow, but some should respond fast. When the service using RestTemplate is used by ~20 threads in parallel then RestTemplate happens to hang and wait for something (some requests to fast endpoints are executed much slower than they should). When I switched to Jersey HTTP client the problem disappeared, so it must be something with RestTemplate itself.

I create the bean with RestTemplateBuilder:

@Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
    return builder.build();
}

And use exchange method for sending requests:

ResponseEntity<Void> result = restTemplate.exchange(url, HttpMethod.POST, request, Void.class);

Does anyone know why RestTemplate can behave in such a way?

Perhaps I misunderstand your question, but this might help: https://stackoverflow.com/questions/21145558/resttemplate-should-be-static-globally-declared — f1dave, Feb 19 '18 at 07:14
You can try using AsyncRestTemplate in-place RestTemplate for concurrent request. — Yogi, Feb 19 '18 at 09:44

score 3 · Answer 1 · answered Oct 23 '20 at 14:09

If you have actuator dependency in the spring boot project, it internally uses Micrometer to generate metrics.

One of the metric is http.client.requests which keep track of all requests handled by RestTemplate. Internally, micrometer creates a ConcurrentHashMap which keeps track of every endpoint as a key. For example- if you have a endpoint say

/api/user/{userId}

and If you hit it 100 times with different userIds, micrometer will create 100 entries in its map. This is not an issue for lower envrieonment. But may cause significant performance degradation in PROD environment where thousands of entries can pile up and slows down the response time before Major GC kicks in.

One solution would be to turn off this metric by including below property in the application.properties

management.metrics.enable.http.client.requests=false

score 2 · Answer 2 · answered Feb 19 '18 at 16:23

It might be related to RestTemplate configuration settings.

You can run netstat -an | grep <app port> (8800 for instance) to look for something like:

...
lot of:
TCP 127.0.0.1:61186 127.0.0.1:8800 TIME_WAIT
TCP 127.0.0.1:61190 127.0.0.1:8800 TIME_WAIT


handful of:
TCP 127.0.0.1:61198 127.0.0.1:8800 ESTABLISHED
TCP 127.0.0.1:61204 127.0.0.1:8800 ESTABLISHED
...

Even though RestTemplate might have gotten configured for a number of connections, RestTemplate's maxPerRoutes prevent a host (with higher response times) from hijacking the conn pool.

You would probably need a more than one RestTemplate instance, going async, fix / improve response times of dependent endpoints, ...

Sometime ago I blog about this issue: Troubleshooting Spring's RestTemplate Requests Timeout

Why RestTemplate hangs when called concurrently?

2 Answers2