Why does this Java/Groovy code cause heap memory exceptions?

This 3rd party script keeps causing heap memory exceptions:

byte[] forwardMessage(byte[] content) {
    s = new Socket("172.17.0.30", 10001);
    s.withStreams {InputStream input, OutputStream output ->
        output.write content
        return readRtsData(input)
    }
}

byte[] readRtsData(input) {
    def vplEndByte = 0xff

    def inStream = new BufferedInputStream(input)

    def bytes = []
    while (bytes.isEmpty() || bytes.last() != vplEndByte) {
        bytes.add(inStream.read())
    }

    bytes
}

That part of the script, which receives messages through TCP/IP after receiving the message, causes the following exception:

Exception in thread “Thread-2” org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:92) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:880) at groovy.lang.Closure.call(Closure.java:279) at groovy.lang.Closure.call(Closure.java:292) at org.codehaus.groovy.runtime.DefaultGroovyMethods$6.run(DefaultGroovyMethods.java:11563) at java.lang.Thread.run(Thread.java:636) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2746) at java.util.ArrayList.ensureCapacity(ArrayList.java:187) at java.util.ArrayList.add(ArrayList.java:378) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at RTSGatewayServer.readRtsData(RTSGatewayServer.groovy:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:880) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151) at RTSGatewayServer$_forwardMessage_closure2.doCall(RTSGatewayServer.groovy:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:880) at groovy.lang.Closure.call(Closure.java:279) at org.codehaus.groovy.runtime.DefaultGroovyMethods.withStreams(DefaultGroovyMethods.java:11462) at org.codehaus.groovy.runtime.dgm$658.invoke(Unknown Source)

I’m guessing there is a better more memory efficient way then using bytes.add(...)?

If anybody can compare the result to what would happen in .NET that would be even better as I’m a .NET developer.

Answer

So the script keeps reading and storing bytes until it sees 0xff.

This seems to be a flawed design. No matter how you tune the JVM you potentially eventually run out of memry. If the remote service chooses to send peta-bytes of deta before that 0xff then you’re going to exhaust memory. My take is that you should always assume that the other participant may be broken or anti-social.

I would therefore set some upper bound on how much data I am prepared to accept. Then, if possible deal with the chunk you’ve receieved and go back and get the next chunk, or if it’s not possible to process in chunks politely put out an error message and stop.

Bottom line: sanitise your inputs. Allowing an outside process to exhaust your memory is a bad thing. Don’t believe them when they say “it can’t happen”.