OutOfMemoryException caused by large Strings

Question

Im using using following Method to catch Data from a webapi:

public static String sendRequest(String requestURL, String data)
        throws IOException {
    URL url = new URL(requestURL + "?" + data);
    URLConnection conn = url.openConnection();
    conn.setReadTimeout(10000);
    BufferedReader in = new BufferedReader(new InputStreamReader(
            conn.getInputStream()));

    String inputLine;
    StringBuilder answerBuilder = new StringBuilder("");
    try {
        while ((inputLine = in.readLine()) != null)
            answerBuilder.append(inputLine);
        in.close();
    } catch (Exception e) {
    }

    return answerBuilder.toString();
}

With some requests, this leads to a OutOfMemoryError caused by a too small HeapSize:

(...)Caused by: java.lang.OutOfMemoryError: (Heap Size=17927KB, Allocated=14191KB, Bitmap Size=2589KB)
at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:95)
at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:132)
at java.lang.StringBuilder.append(StringBuilder.java:272)
at java.io.BufferedReader.readLine(BufferedReader.java:423)
at com.elophant.utils.HTTPUtils.sendRequest(HTTPUtils.java:23)
at (..)

I already swapped from normal String operations like String answer += inputLine to a StringBuilder but this didnt help. How can i solve this Problem? Increasing maximum heap size via export JVM_ARGS="-Xmx1024m -XX:MaxPermSize=256m" isnt an option as its an android app.

what are you trying to download? Is it a entire binary bin file, xml, json? — Francisco Spaeth, Apr 26 '13 at 13:36
Your application does not scale, so cannot solve the problem without rewriting. Of course, you can add more memory, but it will still fail sooner or later. Write the content to into a persistent storage (e.g. filesystem). It might be possible to solve this if you *really* know the maximum possible size of the HTTP resource. — home, Apr 26 '13 at 13:40

score 3 · Answer 1 · answered Apr 26 '13 at 13:39

3

Use a file for temporary storage like when a hard drive starts paging because it's out of memory.

answered Apr 26 '13 at 13:39

Pharap

3,826
5
37
51

Francisco Spaeth · Accepted Answer · 2013-04-26T14:10:25.653

One solution would be to persist the content being downloaded to a storage.

Depending on what you are download you could parse it during its read and store it in a SQL Lite DataBase. This would allow you to use Query language to handle data afterwards. This would be really useful if file being downloaded is a JSON or XML.

In JSON you could get the InputStream as you already do and read stream with the JSON Reader. For every record read from the JSON you can store in a table (or more tables depending on how each record is structured). The good thing from this approach is that at the end you don't need file handling and you already have your data distributed in tables within your database ready to be queried.

score 1 · Answer 3 · answered Apr 26 '13 at 13:39

1

you should write the stringbuilder content into a file and clear it from time to time.

answered Apr 26 '13 at 13:39

Philipp Sander

10,139
6
45
78

LionC · Answer 4 · 2015-10-13T14:52:15.010

1

If your String is really that large, you will need to store it in a file temporarily and process it in chunks (or handle it in chunks while you receive it)

edited Oct 13 '15 at 14:52

answered Apr 26 '13 at 13:40

LionC

3,106
1
22
31

jn1kk · Answer 5 · 2013-04-26T13:58:20.640

0

Not for the faint-hearted, but write your own MyString class that uses a byte for each char~ 50% memory savings! And consequnetely, MyStringBuilder class. Only assuming you are dealing with ASCII.

edited Apr 26 '13 at 13:58

answered Apr 26 '13 at 13:45

jn1kk

5,012
2
45
72

OutOfMemoryException caused by large Strings

5 Answers5

Linked