How to handle the weblogic Too many open files exceptions in Linux environment.
Exception:
<Jan 9, 2018 11:06:01 AM IST> <Notice> <Server> <mwinventory.in> <mwiserver2> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1515557161382> <BEA-002614> <Reopening channel "Default" on 192.16.8.34:13004.> ####<Jan 9, 2018 11:06:01 AM IST> <Error> <Server> <mwinventory.in> <mwiserver2> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1515557161383> <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 192.16.8.34 might be incorrect or another process is using port 13004: java.net.SocketException: Too many open files.>
Root Cause Analysis and Solution:
Too many open files error mean that your JVM is not able to open any new file for processing because of the ulimit constraints on the box
How to check the ulimit set for your box
ulimit is and being set per userid, so please switch to the user in which weblogic processes are running. Let's say weblogic is the username
sudo su - weblogic (or) su - weblogic
Now execute the following command for the weblogic user account
aksarav@mwinventory# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 5289
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 7200
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 515638
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
in the output, open files indicate the maximum open file your user account could possibly hold. you can get this info directly using the command ulimit -n
How many open files your JVM is having
To see how many files are being held by your JVM, First get the PID of your JVM using ps -feww (or) ps -eafand use the following command
/usr/sbin/lsof -p <pid> |wc -l
in the command, replace <pid> with your actual jvm pid
How many open files your useracount is having
To see how many files are being held by your useraccount ( in my case weblogicusr), use the following command
/usr/sbin/lsof -u <username> |wc -l
in the command, replace <username> with your actual user account name, in my case its weblogicusr
After confirming the open file count you can restart the server as a temporary workaround. the permanent solution could be anything from the following
- Analyze the opened files and see why the JVM is holding all these files, make sure that there is no bug in the deployed application which is leading to massive file opening and make sure the code is closing the files properly after opening.
- when you find no issues at the code, you can consider increasing the open file limit
How to increase the openfile limit for the user(weblogicusr)
To increase the open file limit for the user account (weblogicusr) you can perform the following configuration changes in the server
Admin privileges (root) is mandatory to perform the below step
Add the following lines in the end of /etc/security/limits.conf file
weblogicusr soft nofile 9384 weblogicusr hard nofile 9384
here 9384 indicates the maximum openfile limit, replace 9384 with the value as per requirement.
Hope it helps.
Thanks,
A K S A R A V
Write to us at : [email protected]
Follow us on Facebook
To Join our Community in Whatsapp – Click here
