Connection reset while streaming xml

Question

My code needs to download a big xml file (500MB) inside a GZIPInputStream and process it doing some operations for every object. Those operations take time to be completed, and I have many objects to process. I'm using commons http-client 3.1 and stax.

public void download(String url) throws HttpException, IOException, 
                XMLStreamException, FactoryConfigurationError {

        GetMethod getMethod = new GetMethod(url);
        try {
            httpClient.executeMethod(getMethod);    
            Header contentEncoding = getMethod.getResponseHeader("Content-Encoding");
            if (contentEncoding != null) {
                String acceptEncodingValue = contentEncoding.getValue();
                if (acceptEncodingValue.indexOf("gzip") != -1) {
                    processStream(new GZIPInputStream(getMethod.getResponseBodyAsStream()));
                    return;
                }
            }

            processStream(getMethod.getResponseBodyAsStream());
            return;           
        } finally {
            getMethod.releaseConnection();
        }
    }

    protected void processStream(InputStream inputStream) throws XMLStreamException, FactoryConfigurationError {
        XMLStreamReader xmlStreamReader = XMLInputFactory.newFactory().createXMLStreamReader(inputStream);
        //parses xml with Stax           
        //executes some long operations for each object
    }

When I run the code it works till, after two or three hours, I get a SocketException: Connection reset. Looks like the server has closed the connection, is it correct? Is there a way to avoid this error without any change on server-side? If not, how can I deal with it to avoid re-running my application from the beginning?

com.ctc.wstx.exc.WstxIOException: Connection reset
    at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
    at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
    .................
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:168)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:182)
    at java.io.FilterInputStream.read(FilterInputStream.java:116)
    at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
    at java.io.FilterInputStream.read(FilterInputStream.java:90)
    at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
    at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
    at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
    at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
    at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
    at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1034)
    at com.ctc.wstx.sr.StreamScanner.getNextChar(StreamScanner.java:794)
    at com.ctc.wstx.sr.BasicStreamReader.parseNormalizedAttrValue(BasicStreamReader.java:1900)
    at com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:3037)
    at com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2936)
    at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2848)
    at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)

This http://stackoverflow.com/a/3414495/32090 may help – Boris Pavlović Dec 15 '11 at 14:13 — Boris Pavlović, Dec 15 '11 at 14:13

score 0 · Accepted Answer · answered Dec 15 '11 at 14:31

0

One suggestion would be to cache the file locally and then process it later.

ie. your handler simply reads the stream and writes it to a temp file on disk. Then it closes the stream and processes the data from the temp file.

This is probably a good approach anyway as, even if you can keep the link up, the possibilities of some network outage, reduced QoS and so on may make retrieving the file unreliable. You might also be preventing the server from updating it for the entire duration of your processing, which is a bit anti-social.

answered Dec 15 '11 at 14:31

AndyT

1,413
9
11

Thanks for your answer. Good hint, but are you sure the httpclient saves the entire file to disk? I don't think it works this way, maybe it saves some fragments? – javanna Dec 15 '11 at 15:24
No, I'm suggesting you explicitly stream the file to disk: create a new temp file, open an output stream and copy the gzip input stream out to the file. – AndyT Dec 15 '11 at 17:53
I understood. This way I'll have a longer wait at the beginning while saving the entire file, but after no socketexception at all. Right? I was just saying that it isn't exactly the same as the httpclient works, but this would save me! – javanna Dec 15 '11 at 23:18

score 0 · Answer 2 · answered Dec 16 '11 at 21:14

0

If you cannot copy the xml to you local computer try to see if the connection timed out. Maybe the processing of the xml takes too long and the connection gets reset by one of the intermediate servers

answered Dec 16 '11 at 21:14

Yuval Zilberstein

175
1
4
11

Connection reset while streaming xml

2 Answers2