Send an upload in chunks to avoid OutOfMemoryError with httpclient 3.1 and URLConnection

Question

I'm trying to send a file to an IIS server using apache commons httpclient 3.1 but I get out of memory error.

    InputStream is = FileService.getInputStream(fileName, FileService.HDD);
    ByteArrayOutputStream fileToUpload = new ByteArrayOutputStream();
    ByteArrayPartSource file = null;

    try {
        byte[] buffer = new byte[1024];
        for (int len; (len = is.read(buffer)) != -1;)
            fileToUpload.write(buffer, 0, len);

        file = new ByteArrayPartSource(fileName, fileToUpload.toByteArray());

        post.setContentChunked(false); 
        Part[] part = new Part[] { new FilePart(fileName, file)}; 
        post.setRequestEntity(new MultipartRequestEntity(part, post.getParams())); 
    }
    catch (OutOfMemoryError e) {
    }
    finally {
        try {
            if (is != null) is.close();
            if (fileToUpload != null) fileToUpload.close();
        } catch (IOException e) {
        }
    }

OutOfMemoryError occurs at fileToUpload.toByteArray(). Is there any way to send file in chunks? I can't use file object because I get permission errors.

I also tried without using httpClient but I still get out of memory error

    InputStream fileToUpload = FileService.getInputStream(fileName, FileService.HDD);
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = null;

    try {
        writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

        writer.println("--" + boundary);
        writer.println("Content-Disposition: form-data; name=\"binaryFile\"; filename=\""+ fileName + "\"");
        writer.println("Content-Type: application/octet-stream");
        writer.println("Content-Transfer-Encoding: binary");
        writer.println();
        writer.flush();

        LogService.info("FileSender#setFileToUpload() Before");
        byte[] buffer = new byte[2048];
        int bytesRead;
        while ((bytesRead = fileToUpload.read(buffer)) != -1){
            output.write(buffer, 0, bytesRead);
        }

        LogService.info("FileSender#setFileToUpload() After");

        output.flush();
        writer.println();
        writer.flush();
        writer.println("--" + boundary + "--");

    } finally {
         if (writer != null) writer.close();
         if (output != null)  output.close();
         if (fileToUpload != null) fileToUpload.close();
    }

http://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms — Pratik Shelar, Aug 01 '14 at 03:41
I am not allowed to change the settings of JVM since I'm not the only one using it. — user3814700, Aug 01 '14 at 04:50

score 0 · Answer 1 · answered Aug 01 '14 at 03:47

0

You need to stream the data, so it's not all in memory at once something like:

    InputStream is = FileService.getInputStream(fileName, FileService.HDD);
    try {
        InputStreamBody inputStreamBody = new InputStreamBody(is, fileName);
        MultipartEntity entity = new MultipartEntity();
        entity.addPart(fileName, inputStreamBody);
        post.setEntity(entity);

        httpClient.execute(post);
        ...
    } finally {
        is.close();
    }

answered Aug 01 '14 at 03:47

Tom

43,583
4
41
61

InputStreamBody is not supported in HttpClient 3.1. It is only in HttpClient 4.1 and I'm restricted to JDK 1.4 so I could only use HttpClient 3.1. – user3814700 Aug 01 '14 at 04:49
Dang! I'll leave this for anyone searching without those constraints - I've created a second answer which may make your non-httpclient code work. – Tom Aug 04 '14 at 06:01
@user3814700 So upgrade. The key to not running out of memory is not to load entire files *into* memory. – user207421 Aug 04 '14 at 07:00

Tom · Answer 2 · 2014-08-04T06:05:19.230

In your non-httpclient version, try moving the output.flush() so you're sending the chunks you read in over the wire as you read them rather than buffering up the entire file in memory before flushing it (you may want to experiment with other chunk sizes):

    InputStream fileToUpload = FileService.getInputStream(fileName, FileService.HDD);
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = null;

    try {
        writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

        writer.println("--" + boundary);
        writer.println("Content-Disposition: form-data; name=\"binaryFile\"; filename=\""+ fileName + "\"");
        writer.println("Content-Type: application/octet-stream");
        writer.println("Content-Transfer-Encoding: binary");
        writer.println();
        writer.flush();

        LogService.info("FileSender#setFileToUpload() Before");
        byte[] buffer = new byte[2048];
        int bytesRead;
        while ((bytesRead = fileToUpload.read(buffer)) != -1){
            output.write(buffer, 0, bytesRead);
            output.flush();// <-----------------------------------
        }//                                                      |
        //                                                       |
        LogService.info("FileSender#setFileToUpload() After");// |
        //                                                       |
        // -------------------------------------------------------
        writer.println();
        writer.flush();
        writer.println("--" + boundary + "--");

    } finally {
         if (writer != null) writer.close();
         if (output != null)  output.close();
         if (fileToUpload != null) fileToUpload.close();
    }

I already tried this but based on what I found in the web, URLConnection will buffer data to the memory to set Content-Length header. So even if I flush the output stream, I will still get OutOfMemoryError. — user3814700, Aug 05 '14 at 05:26

Send an upload in chunks to avoid OutOfMemoryError with httpclient 3.1 and URLConnection

2 Answers2