Why does Java throw java.lang.OutOfMemoryError: Metaspace but there is plenty of free metaspace?

In my application I’m creating lots of java classes at runtime with javassist library. At some point a java.lang.OutOfMemoryError: Metaspace is thrown but java process monitoring (based on java.lang.management.MemoryPoolMXBean) reports that there is plenty of free metaspace. Why is that? And how to use 100% of metaspace memory pool?

I’ve created a minimal application that reproduces the problem https://github.com/vlkv/java_metaspace_oom

Donwload it, cd to the project dir and then execute ‘ant run’. In this app, I’ve set -XX:MaxMetaspaceSize=100m, but OOM is thrown at some point around 23564Kb of metaspace used.

Call stack of the error is:

javassist.CannotCompileException: by java.lang.OutOfMemoryError: Metaspace
    at javassist.ClassPool.toClass(ClassPool.java:1170)
    at javassist.CtClass.toClass(CtClass.java:1316)
    at com.tradingview.Main.generateRandomClass(Main.java:53)
    at com.tradingview.Main.main(Main.java:24)
Caused by: java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
    at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javassist.ClassPool.toClass2(ClassPool.java:1183)
    at javassist.ClassPool.toClass(ClassPool.java:1164)
    ... 3 more

Answer

Wow, I’ve just found an answer. The problem is that in my application I create a separate new ClassLoader for every generated Class. I’ve done a modification in my sample app that puts all the generated classes to one ClassLoader and after that, OOM is thrown at the point when free metaspace is almost zero. The fix is here https://github.com/vlkv/java_metaspace_oom/commit/d6bcade51f79758e2413d1852c771f163392c294 branch fix_of_the_problem

Does somebody knows about what are the limits for class loaders in java process? And how to increase them?

Also found this Why MetaSpace Size is twice as big as Used MetaSpace?