What’s the impact of large number of file descriptors on Java applications

We have a web application deployed on Tomcat, which can have over 300k file descriptors, while our limit for single process is 250k. The strange thing is: when the server is running, the number goes up (400k) and down (100k). Sometimes, we cannot ssh into the OS when the number is high. However, we didn’t find any error related to many open files or socket establish problem. Majority of the file descriptors are related to jar files loaded by JVM. My questions are :
1. How does OS (CentOS 7) count the file descriptors for tomcat? I don’t think Tomcat keeps those files open while its running. 2. Why is the number not fixed? Instead, there are lots of duplicate jar files. 3. Is it normal to have so many file descriptors?

Answer

Operating systems have a limit on the number of files that can be concurrently open by any one process. The default for most distributions is only 1024 files. Each open file also has an associated file-descriptor. Socket connections are treated like files and they use file descriptor, and are therefore subject to the same resource limits.

1. You can verify or change the maximum limit by the command ulimit.

2. You can also view value of the MBean Attibutes – MaxFileDescriptorCount & OpenFileDescriptorCount by running the JMX tool- JConsole.

3. When OpenFileDescriptorCount is less than MaxFileDescriptorCount your application works fine, otherwise you would get java.io.IOException: Too many open files which causes malfunctioning of your application .

4. Normally for an application the count of the FD(File Descriptor) goes up/down to certain level. But it should be within MaxFileDescriptorCount.