With Spring Boot, I'm not entirely sure why this was necessary (I got the /error fallback even though @ResponseBody was defined on an @ExceptionHandler), but the following in itself did not work:
@ResponseBody
@ResponseStatus(HttpStatus.BAD_REQUEST)
@ExceptionHandler(IllegalArgumentException.class)
public ErrorMessage handleIllegalArguments(HttpServletRequest httpServletRequest, IllegalArgumentException e) {
    log.error("Illegal arguments received.", e);
    ErrorMessage errorMessage = new ErrorMessage();
    errorMessage.code = 400;
    errorMessage.message = e.getMessage();
    return errorMessage;
}
It still threw an exception, apparently because no producible media types were defined as a request attribute:
// AbstractMessageConverterMethodProcessor
@SuppressWarnings("unchecked")
protected <T> void writeWithMessageConverters(T value, MethodParameter returnType,
        ServletServerHttpRequest inputMessage, ServletServerHttpResponse outputMessage)
        throws IOException, HttpMediaTypeNotAcceptableException, HttpMessageNotWritableException {
    Class<?> valueType = getReturnValueType(value, returnType);
    Type declaredType = getGenericType(returnType);
    HttpServletRequest request = inputMessage.getServletRequest();
    List<MediaType> requestedMediaTypes = getAcceptableMediaTypes(request);
    List<MediaType> producibleMediaTypes = getProducibleMediaTypes(request, valueType, declaredType);
if (value != null && producibleMediaTypes.isEmpty()) {
        throw new IllegalArgumentException("No converter found for return value of type: " + valueType);   // <-- throws
    }
// ....
@SuppressWarnings("unchecked")
protected List<MediaType> getProducibleMediaTypes(HttpServletRequest request, Class<?> valueClass, Type declaredType) {
    Set<MediaType> mediaTypes = (Set<MediaType>) request.getAttribute(HandlerMapping.PRODUCIBLE_MEDIA_TYPES_ATTRIBUTE);
    if (!CollectionUtils.isEmpty(mediaTypes)) {
        return new ArrayList<MediaType>(mediaTypes);
So I added them.
@ResponseBody
@ResponseStatus(HttpStatus.BAD_REQUEST)
@ExceptionHandler(IllegalArgumentException.class)
public ErrorMessage handleIllegalArguments(HttpServletRequest httpServletRequest, IllegalArgumentException e) {
    Set<MediaType> mediaTypes = new HashSet<>();
    mediaTypes.add(MediaType.APPLICATION_JSON_UTF8);
    httpServletRequest.setAttribute(HandlerMapping.PRODUCIBLE_MEDIA_TYPES_ATTRIBUTE, mediaTypes);
    log.error("Illegal arguments received.", e);
    ErrorMessage errorMessage = new ErrorMessage();
    errorMessage.code = 400;
    errorMessage.message = e.getMessage();
    return errorMessage;
}
And this got me through to have a "supported compatible media type", but then it still didn't work, because my ErrorMessage was faulty:
public class ErrorMessage {
    int code;
    String message;
}
JacksonMapper did not handle it as "convertable", so I had to add getters/setters, and I also added @JsonProperty annotation
public class ErrorMessage {
    @JsonProperty("code")
    private int code;
    @JsonProperty("message")
    private String message;
    public int getCode() {
        return code;
    }
    public void setCode(int code) {
        this.code = code;
    }
    public String getMessage() {
        return message;
    }
    public void setMessage(String message) {
        this.message = message;
    }
}
Then I received my message as intended
{"code":400,"message":"An \"url\" parameter must be defined."}