We have a struts2 web application running in Tomcat 7 that is intermittently experiencing the same problem:
- Application operating perfectly normally
- After a variable amount of time (anywhere from 10 minutes to 7 days) the application slows down to a halt
- Multiple java.lang.OutOfMemory errors in catalina.out
- Linux free -m shows all physical and swap memory is used up (even though I’ve set Xmx8G in setenv.sh)
I’ve configured Tomcat to create an automatic heap dump on OutOfMemory and when I analyse the dump using either jmap or MAT in Eclipse every dump shows the same thing. All the memory is taken up by nearly 1 million java.util.LinkedList$Node’s that seem to be generated to by PageOutputStream.
As these nodes seem to be generated by PageOutputStream which is part of page rendering in Struts my thought is that there is an infinite loop of Struts tags in one of our JSP pages that’s creating all these linked list nodes but I’m struggling to understand how I might find out what that page might be.
I’m wondering if anyone knows of a method to isolate this problem further either down to a line of code or a jsp page?
Advertisement
Answer
I eventually found out that the problem was a Struts tag in the JSP that was mangled. I was retrieving a set of results in the controller and displaying them in the JSP using <s:iterator>
These search results included some embedded HTML which created a much longer iteration, not infinite but long enough to consume memory quickly. Once I fixed the embedded HTML the problem disappeared.