HttpURLConnection is not returning all headers

Question

I'm printing HTTP headers using the following code.

    URL url = new Url("htttp://example.com/example");
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();

    Map<String, List<String>> map = conn.getHeaderFields();
    for (Map.Entry<String, List<String>> entry : map.entrySet()) {
        System.out.println("Key : " + entry.getKey() +
                " ,Value : " + entry.getValue());
    }

Here's the output :

Key : null ,Value : [HTTP/1.1 200 OK]
Key : Accept-Ranges ,Value : [bytes]
Key : Cache-Control ,Value : [max-age=604800, public]
Key : Connection ,Value : [Keep-Alive]
Key : Date ,Value : [Mon, 03 Oct 2016 18:01:06 GMT]
Key : ETag ,Value : ["159eb4-53dce1f957880-gzip"]
Key : Expires ,Value : [Mon, 10 Oct 2016 18:01:06 GMT]
Key : Keep-Alive ,Value : [timeout=5, max=100]
Key : Last-Modified ,Value : [Sat, 01 Oct 2016 13:59:46 GMT]
Key : Server ,Value : [Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4]
Key : Transfer-Encoding ,Value : [chunked]
Key : Vary ,Value : [Accept-Encoding,User-Agent]

Now I'm trying via curl:

$ curl -v -D - http://example.com/example -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 111.111.111.111...
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to example.com (111.111.111.111) port 80 (#0)
> GET example.com/example HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 03 Oct 2016 18:25:11 GMT
Date: Mon, 03 Oct 2016 18:25:11 GMT
< Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
< Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
< ETag: "159eb4-53dce1f957880"
ETag: "159eb4-53dce1f957880"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 1416884
Content-Length: 1416884
< Cache-Control: max-age=604800, public
Cache-Control: max-age=604800, public
< Expires: Mon, 10 Oct 2016 18:25:11 GMT
Expires: Mon, 10 Oct 2016 18:25:11 GMT
< Vary: Accept-Encoding,User-Agent
Vary: Accept-Encoding,User-Agent

< 
{ [1080 bytes data]
 30 1383k   30  427k    0     0   116k      0  0:00:11  0:00:03  0:00:08  116k^C

See how the Content-Length header is missing in the java output? Why does this happen? How can I fix this?

It's provided via an API. There is no fix and you don't need one. — user207421, Oct 03 '16 at 18:34
@EJP How come I don't need a fix? Shouldn't the API return `Content-Length` if the server sends a `Content-Length`? — Binoy Babu, Oct 03 '16 at 18:38
The API I refer to is `getContentLength()` (and `getContentLengthLong()`). — user207421, Oct 03 '16 at 18:46
I can reproduce your bug with `www.google.com`, the header is visible in curl but not with the Java code — Nicolas Filotto, Oct 03 '16 at 18:56
@NicolasFilotto It isn't a bug yet. Can you show the header was sent in the Java case? — user207421, Oct 03 '16 at 18:58
@EJP in my case it was because `curl` doesn't follow the redirect while the java code does by default so by calling `conn.setInstanceFollowRedirects(false)` , I finally get the content length in the list of my headers — Nicolas Filotto, Oct 03 '16 at 19:03
Try to call `conn.setInstanceFollowRedirects(false)` to see if it helps — Nicolas Filotto, Oct 03 '16 at 19:04
Have you added conn.setInstanceFollowRedirects(false) just after HttpURLConnection conn = (HttpURLConnection) url.openConnection();? — Nicolas Filotto, Oct 03 '16 at 19:18

score 4 · Answer 1 · edited Oct 07 '21 at 08:57

4

Content-Length is missing because the server sent the response in chunks. It's the expected behavior. Look at the Transfer-Encoding header in the Java response, and here's RFC 2616, 4.4 Message Length (see the 3rd item of the list).

edited Oct 07 '21 at 08:57

Community

1
1

answered Oct 03 '16 at 18:59

nandsito

3,782
2
19
26

So how do I disable this? – Binoy Babu Oct 03 '16 at 19:15
@BinoyBabu i'm afraid that's not possible. See this [related question](http://stackoverflow.com/questions/18174435/avoiding-chunked-encoding-of-http-1-1-response). [RFC 2616, 3.6.1](https://tools.ietf.org/html/rfc2616#section-3.6.1) says `All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding`, so i think you'll have to cope with this header – nandsito Oct 03 '16 at 19:52
@BinoyBabu [RFC 2616, 14.41](https://tools.ietf.org/html/rfc2616#section-14.41) says `Many older HTTP/1.0 applications do not understand the Transfer-Encoding header`, so if you can configure the server and really wish to disable `Transfer-Encoding`, you can roll your server back to HTTP/1.0, but i strongly discourage it for you'll lose much of the 1.1 improvementes, such as `Transfer-Encoding` itself – nandsito Oct 03 '16 at 19:59
Are you saying that it's impossible to find the content length from headers in HTTP 1.1? I have seen many programs doing this. – Binoy Babu Oct 03 '16 at 20:21
@BinoyBabu instead of observing the `Content-Length` header you'll have to find out the entity-body size by other means, e.g. downloading everything and counting the number of octets after that – nandsito Oct 03 '16 at 20:29
@BinoyBabu to show download progress you'll have to do some server configuring to ensure that it doesn't split the response in chunks, so the response is sent as a whole with `Content-Length`. Or else it won't be possible to make the download progress – nandsito Oct 03 '16 at 20:44
@BinoyBabu try to inspect the request headers wget and curl send to the server. Maybe there's a hint there – nandsito Oct 03 '16 at 20:58

score 2 · Answer 2 · answered Oct 03 '16 at 18:43

2

The 'Content-Length' is parsed automatically and can be retrieved via conn.getContentLength() or conn.getContentLengthLong(). See https://docs.oracle.com/javase/7/docs/api/java/net/URLConnection.html#getContentLengthLong() for more info.

answered Oct 03 '16 at 18:43

Mike Laren

8,028
17
51
70

I doesn't explain why the `content-length` is not in the headers, this will return `-1` if the header doesn't exist which is the case here – Nicolas Filotto Oct 03 '16 at 18:51
@NicolasFilotto It will return -1 if the header wasn't *sent*, which *isn't* the case here, according to the `curl` output. – user207421 Oct 03 '16 at 18:52
@EJP no I have just tested with `www.google.com` with curl I see that `Content-Length` is 258 but with java this header is not part of the list so I get -1 when I call this method – Nicolas Filotto Oct 03 '16 at 18:55
@NicolasFilotto So it wasn't sent for some reason in the Java case. – user207421 Oct 03 '16 at 18:55

score 0 · Answer 3 · answered Oct 03 '16 at 19:28

The response in case of java includes 'transfer-encoding' header which is missing in case of the curl response. Also when I tried it, I got a 404 (not found) for my attempt. I get a 404 when I try it from my browser as well.

My guess is that you have an overridden host entry for example.com that loops back to your own tomcat or similar. Its an issue with your environment.

This is what I get :

    /**
     *
     * <P> jdk-1.8 </P>
     * <PRE>
Key :null || Values :[HTTP/1.1 404 Not Found]
Key :X-Cache || Values :[HIT]
Key :Server || Values :[ECS (ewr/1445)]
Key :Etag || Values :["359670651+gzip+ident"]
Key :Cache-Control || Values :[max-age=604800]
Key :x-ec-custom-error || Values :[1]
Key :Vary || Values :[Accept-Encoding]
Key :Last-Modified || Values :[Fri, 09 Aug 2013 23:54:35 GMT]
Key :Expires || Values :[Mon, 10 Oct 2016 19:20:29 GMT]
Key :Content-Length || Values :[1270]
Key :Date || Values :[Mon, 03 Oct 2016 19:20:29 GMT]
Key :Content-Type || Values :[text/html]
     * </PRE>
     * @param urlStr (http://example.com/example)
     */
    private static void printHeaders(String urlStr) {

        try {
            URL urlRef = new URL(urlStr);
            URLConnection urlConnection = urlRef.openConnection();
            Map<String, List<String>> headerFields = urlConnection.getHeaderFields();

            for (String headerName : headerFields.keySet()) {
                List<String> headerEntry = headerFields.get(headerName);
                System.out.println("Key :"+headerName +" || Values :"+headerEntry);
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

HttpURLConnection is not returning all headers

3 Answers3