android java not enough sequential memory for stringbuilder

My app is parsing a large http response, the http response is over 6 megabytes and is json, but not in a standard schema.

        final char[] buffer = new char[0x10000];
        StringBuilder out = new StringBuilder();
        Reader in = new InputStreamReader(is, "UTF-8");
        int read;
        System.gc();
        do
        {
            read = in.read(buffer, 0, buffer.length);
            if (read > 0)
            {
                out.append(buffer, 0, read);
            }
        } while (read >= 0);
        in.close();
        is.close();
        in = null;
        is = null;
        System.gc();
        return out.toString();

JavaScript
​x
 
        final char[] buffer = new char[0x10000];        StringBuilder out = new StringBuilder();        Reader in = new InputStreamReader(is, "UTF-8");        int read;        System.gc();        do        {            read = in.read(buffer, 0, buffer.length);            if (read > 0)            {                out.append(buffer, 0, read);            }        } while (read >= 0);        in.close();        is.close();        in = null;        is = null;        System.gc();        return out.toString();​

It doesn’t matter if there is a bufferedreader from a file, or an inputstream, the StringBuilder simply cannot contain the entire object and it fails at out.append(buffer, 0, read); or it will fail at out.toString() as another copy may be made

IOUtils.copy from the apache library is doing the same things under the hood and it will also fail.

How can I read this large object in for further manipulation. Right now this method fails on Android 2.2 and 2.3 devices, and uses more memory than I want on newer devices.

Similar questions all have answers that involve appending to a stringbuilder, reading in lines, or have incomplete solutions that are only hints, and that doesn’t work.

Answer

You need to do one of two things:

Get multiple smaller JSON responses from the server and parse those. This might be preferable on a mobile device, as large chunks of data might not be transmitted reliably, which will cause the device to request the entire thing repeatedly.
Use a streaming JSON parser, such as Jackson, to process the data as it comes in.

Advertisement

Answer