I recently found that one of my services was taking a large amount of time in garbage collection because the max heap size was too small. The service was there before I started, so I was unaware of the low heap size. I would like to set an alarm to warn me if it goes back up past a certain point, but I also do not want to give it more resources than it actually needs. What do you think is a reasonable level to be alarmed about for garbage collection and percent heap used?
I was thinking that an alarm on average heap usage would be ~85 percent, and 100 ms of gc / 5 min.
I understand this is based on requirements and hardware, but I am really looking for some benchmark or standard upon which to make my decision.
Advertisement
Answer
Alex Lockwood’s answer says this:
The recommended “max level” of heap memory usage and GC time is as little as possible.
That is misleading. I’d actually advise the reverse. It is a bad idea to try to squeeze the heap size, because that will lead to your application running the GC more frequently, and spending less time (on average) doing useful work.
The problem is basically this. The classical (non-concurrent) GC gets run when the JVM runs out of space to allocate objects. It then traverses the non-garbage objects, copying them to a different “space”. The processor time to run a GC cycle is most strongly dependent on the amount of non-garbage … but the useful work it achieves (the amount of space it releases) is proportional to heapsize - nongarbage. So when you squeeze the heapsize, you decrease the amount of useful work that the GC does … for the same processor time expenditure.
The original question says this:
I was thinking that an alarm on average heap usage would be ~85 percent, and 100 ms of gc / 5 min.
It is probably not useful to set a monitor / alarm on an absolute level of GC CPU usage. GC time will depend on server activity as well as GC efficiency. You don’t want the GC alarm to go off every time your server gets busy.
An average heap usage of 85% is reasonable level to be alarming at, though once again setting an alarm at a fixed level might give too many false alarms.
An alternative to this is to use the JVM options to set a “percentage time spent in GC” threshold, and combine that with the “kill JVM on OutOfMemoryException” option, and put a auto-restart loop in the server’s launch scripts. Then monitor for restarts.