Problems with JSON and Memory

Question

I'm trying to load a lot of data from the web to my android application and I've been getting this error:

07-18 10:16:00.575: E/AndroidRuntime(30117): java.lang.OutOfMemoryError: [memory exhausted]

and already read a lot about JSON. I've found some solutions but nothing really helped me.

This is my code :

public class HistoricoAdapter extends BaseAdapter {
    private Context ctx;
    JSONArray jsonArray;

    public HistoricoAdapter(Context ctx) {
        this.ctx = ctx;

        String readHttp = readHttp();

        try {
            // transforma a string retornada pela função readHttp() em array
            jsonArray = new JSONArray(readHttp);
        } catch (Exception e) {
            e.printStackTrace();
        }

    }

    public String readHttp() {

        // Acessa a URL que retorna uma string com  os dados do banco
        StringBuilder builder = new StringBuilder();
        HttpClient client = new DefaultHttpClient();
        HttpGet httpGet = new HttpGet("some url");
        try {
            HttpResponse response = client.execute(httpGet);
            StatusLine statusLine = response.getStatusLine();

            int statusCode = statusLine.getStatusCode();
            if (statusCode == 200) {
                HttpEntity entity = response.getEntity();
                InputStream content = entity.getContent();
                BufferedReader reader = new BufferedReader(
                        new InputStreamReader(content));
                String line;
                while ((line = reader.readLine()) != null) {
                    builder.append(line);
                }
            } else {
                Log.e(this.toString(), "Erro ao ler JSON!");
            }
        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return builder.toString();
    }

    public int getCount() {
        return jsonArray.length();
    }
    public boolean isEmpty(){

        if(jsonArray.toString().isEmpty()){
            return true;
        }
        else {
            return false;
        }
    }

    public Object getItem(int position) {
        JSONObject ob = null;
        try {
            ob = jsonArray.getJSONObject(position);
        } catch (JSONException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return ob;
    }

    public long getItemId(int arg0) {
        return 0;
    }

    public View getView(int position, View view, ViewGroup arg2) {

        LayoutInflater layout = (LayoutInflater) ctx
                .getSystemService(Context.LAYOUT_INFLATER_SERVICE);

        View v = layout.inflate(R.layout.listar_compromisso, null);

        try {
            JSONObject obj = (JSONObject) getItem(position);



        } catch (Exception ex) {

        }

        return v;
    }
}

Can anyone predict why am I getting this error?

It is really large,so is there any other way to do it since it is large? — João Pedro Segurado, Jul 18 '14 at 13:37
Does anyone know to import a large data from the web to sqlite? — João Pedro Segurado, Jul 30 '14 at 12:43

meda · Answer 1 · 2014-07-18T13:37:23.543

2

If you get this error then your JSON must be too large to be buffered into memory.

The problem is org.json is too basic to handle that.

You need an advanced library to stream responses instead such as GSON or Jackson.

edited Jul 18 '14 at 13:37

answered Jul 18 '14 at 13:28

meda

45,103
14
92
122

@JoãoPedroSegurado ok I added some links – meda Jul 18 '14 at 13:37
In gson,do I have to install it or something? – João Pedro Segurado Jul 18 '14 at 13:48
I am brazilian and I am having some problems with english,can you help me out,how can I import it? – João Pedro Segurado Jul 18 '14 at 14:00
http://www.techrepublic.com/blog/software-engineer/use-gson-to-work-with-json-in-your-android-apps/ – meda Jul 18 '14 at 14:03
I'm still having a lot of problems,can you give me some example based on my code? – João Pedro Segurado Jul 18 '14 at 18:05
I actually got into gson and jackson, GSON don't give me any error but don't insert all the data into the sqlite,but it does store the data in an array,and jackson gives me error of memory still,any idea of what could I do? – João Pedro Segurado Jul 21 '14 at 22:20
@JoãoPedroSegurado sure if I give you a sample project would it be helpful? im just not sure on how to reproduce a large json – meda Jul 21 '14 at 22:58
It would be very helpful,if you can't reproduce a large json,I can send you a URL who has a large one – João Pedro Segurado Jul 22 '14 at 00:05
@JoãoPedroSegurado ok I got it , by 2morrow I will get back to you – meda Jul 22 '14 at 00:26
I don't sound slack,but did you got something? cos I kind quit to try insertindo to the sqlite and now I'm using the internal storage,but I would like to know about this – João Pedro Segurado Jul 25 '14 at 01:54

score 0 · Answer 2 · answered Jul 18 '14 at 13:41

There are a few issues with this code. First thing I noticed is your not calling your web request inside of an asynctask. You want to use async task for all long running operations and you have to use it for web calls.

AsyncTask must be subclassed to be used. The subclass will override at least one method (doInBackground(Params...), and most often will override a second one

private class DownloadFilesTask extends AsyncTask<URL, Integer, Long> {
 protected Long doInBackground(URL... urls) {
     int count = urls.length;
     long totalSize = 0;
     for (int i = 0; i < count; i++) {
         totalSize += Downloader.downloadFile(urls[i]);
         publishProgress((int) ((i / (float) count) * 100));
         // Escape early if cancel() is called
         if (isCancelled()) break;
     }
     return totalSize;
 }

 protected void onProgressUpdate(Integer... progress) {
     setProgressPercent(progress[0]);
 }

 protected void onPostExecute(Long result) {
     showDialog("Downloaded " + result + " bytes");
 }
}

You should also use JSON to build your object object. It has been tested and you can find information about how much objects can handle under there documentation. Here is a little bit of information from there site:

Gson Performance and Scalability

Here are some metrics that we obtained on a desktop (dual opteron, 8GB RAM, 64-bit Ubuntu) running lots of other things along-with the tests. You can rerun these tests by using the class PerformanceTest. Strings: Deserialized strings of over 25MB without any problems (see disabled_testStringDeserializationPerformance method in PerformanceTest) Large collections: Serialized a collection of 1.4 million objects (see disabled_testLargeCollectionSerialization method in PerformanceTest) Deserialized a collection of 87,000 objects (see disabled_testLargeCollectionDeserialization in PerformanceTest) Gson 1.4 raised the deserialization limit for byte arrays and collection to over 11MB from 80KB.

Also, build your class objects using Json to Java object convert like there one. There are many more if this link doesnt end of working later on. Do a simple Google search.

Convert from json to Java Objects but you can also convert to your own custom objects.

Gson gson = new Gson();
int[] ints = {1, 2, 3, 4, 5};
String[] strings = {"abc", "def", "ghi"};

(Serialization)
gson.toJson(ints);     ==> prints [1,2,3,4,5]
gson.toJson(strings);  ==> prints ["abc", "def", "ghi"]

Problems with JSON and Memory

2 Answers2