To improve troubleshooting on our services, we implemented a mechanism of "Request IDs", in which a unique ID is assigned to each request and then when a service calls another while servicing a particular request, it sends the request ID as an HTTP header, so that if an error occurs in the called service, we can easily find out what was the original request. To implement this feature, we've created two filters:
- a
ContainerRequestFilterwhich reads the ID from the request headers and sets it in theContainerRequestContext - a
ClientRequestFilterthat adds the Request ID to all requests issued by aClient
For this to work, we are registering the ClientRequestFilter on every request, like so:
public class MyServiceClient {
private WebTarget target;
@Inject
public MyServiceClient(Client client, @Context ContainerRequestContext ctx) {
// This is called on every request, with a new context
this.target = client
.target("/target/endpoint")
.register(new RequestIdFilter(ctx)); // custom ClientRequestFilter
}
public void doSomething() {
this.target.path("resource")
.request()
.get();
}
}
This works fine most of the time, but we've encountered sporadic errors which seem to be caused by thread-safety issues. What happens is that if two threads call MyServiceClient::doSomething at the same time, the WebTarget instance will eventually call ClientConfig::initRuntime which at some point collects all Binders configured and calls Binder::bind on them. Now, AbstractBinder::bind is clearly not thread-safe, so sometimes requests fail with:
java.lang.IllegalArgumentException: Recursive configuration call detected.
at org.glassfish.hk2.utilities.binding.AbstractBinder.bind(AbstractBinder.java:179)
at org.glassfish.jersey.model.internal.CommonConfig.configureBinders(CommonConfig.java:676)
at org.glassfish.jersey.model.internal.CommonConfig.configureMetaProviders(CommonConfig.java:641)
at org.glassfish.jersey.client.ClientConfig$State.configureMetaProviders(ClientConfig.java:372)
at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:405)
at org.glassfish.jersey.client.ClientConfig$State.access$000(ClientConfig.java:90)
at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:122)
at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:119)
at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:340)
at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:733)
at org.glassfish.jersey.client.ClientRequest.getConfiguration(ClientRequest.java:286)
at org.glassfish.jersey.client.JerseyInvocation.validateHttpMethodAndEntity(JerseyInvocation.java:135)
at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:105)
at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:101)
at org.glassfish.jersey.client.JerseyInvocation.<init>(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:420)
at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:316)
According to Is java Jersey 2.1 client thread safe?, this problem could have been caused by a non-thread-safe Binder registered by someone else (perhaps JacksonBinder, from Dropwizard). What do you think?
If the approach is wrong, do you know of a better way of implementing this "Request ID" functionality?
Thanks!
Edit 1: Here is the code for the RequestIdFilter
public class RequestIdFilter implements ClientRequestFilter {
private ContainerRequestContext context;
public RequestIdFilter(ContainerRequestContext context) {
this.context = context;
}
@Override
public void filter(ClientRequestContext clientCtx) throws IOException {
String requestId = (String) context.getProperty(X_REQUEST_ID); // This is set by a ContainerRequestFilter
clientCtx.getHeaders().putSingle(X_REQUEST_ID, requestId);
}
}
Edit 2: The issue has been (temporarily) fixed by passing the Request ID as a query parameter when creating the WebTarget in the MyServiceClient constructor. This seems to be thread-safe and avoids registering a new filter on every request.