Getting Symbol table error due to “Too many open files” on Linux

m.mercury.diagnostics.server.persistence.symboltable.impl.LocalSymbolTable.setValue(LocalSymbolTable.java:458)
at com.mercury.diagnostics.server.persistence.symboltable.impl.LocalSymbolTable.setValue(LocalSymbolTable.java:415)
at com.mercury.diagnostics.server.persistence.symboltable.impl.SymbolManager.unsafeGetLocalKey(SymbolManager.java:382)
at com.mercury.diagnostics.server.persistence.symboltable.impl.SymbolManager.getLocalKey(SymbolManager.java:297)
at com.mercury.diagnostics.server.persistence.symboltable.impl.SymbolManager.getNodeDataFromReferenceCache(SymbolManager.java:477)
at com.mercury.opal.mediator.onlinecache.OnlineCache$AggregationContext.createNode(OnlineCache.java:1893)
at com.mercury.diagnostics.common.data.graph.impl_oo.AbstractTreeNode.createChildNode(AbstractTreeNode.java:248)
at com.mercury.diagnostics.common.data.graph.impl_oo.AggregateTreeNode$AggregateChildTreeTraverser.enterNode(AggregateTreeNode.java:363)
at com.mercury.diagnostics.common.data.graph.TreeTraversal.traverseDepthFirst(TreeTraversal.java:88)
at com.mercury.diagnostics.common.data.graph.TreeTraversal.<init>(TreeTraversal.java:45)
at com.mercury.diagnostics.common.data.graph.impl_oo.AggregateTreeNode.aggregateChildren(AggregateTreeNode.java:281)
at com.mercury.diagnostics.common.data.graph.impl_oo.AggregateTreeNode.aggregate(AggregateTreeNode.java:271)
at com.mercury.opal.mediator.onlinecache.OnlineCache.doTrendAggregation(OnlineCache.java:858)
at com.mercury.opal.mediator.onlinecache.OnlineCache.addBucket(OnlineCache.java:661)
at com.mercury.opal.mediator.onlinecache.OnlineCache.addBucket(OnlineCache.java:554)
at com.mercury.opal.mediator.communication.ProbeTrendsPullerTask._updateCache(ProbeTrendsPullerTask.java:324)
at com.mercury.opal.mediator.communication.ProbePullerTask.updateCache(ProbePullerTask.java:1099)
at com.mercury.opal.mediator.communication.ProbeDataPuller$PullerThread.implRun(ProbeDataPuller.java:366)
at com.mercury.diagnostics.common.util.StateRunnable.run(StateRunnable.java:136)
at java.lang.Thread.run(Unknown Source)
Caused by: com.sleepycat.je.log.LogFileNotFoundException: (JE 3.1.0) Couldn’t open file /prod/pophma1/diagnostics/v9.0.1/Server/./archive/mediator-ohmg0008.ohctr.gxs.com/symboltable/000216ac.jdb: /prod/pophma1/diagnostics/v9.0.1/Server/./archive/mediator-ohmg0008.ohctr.gxs.com/symboltable/000216ac.jdb (Too many open files)
at com.sleepycat.je.log.FileManager.makeFileHandle(FileManager.java:676)
at com.sleepycat.je.log.FileManager.access$500(FileManager.java:49)
at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:1652)
… 30 more

The root cause of this issue is due to Linux system running out of file descriptors and so it gives error due to "Too many open files."

The resolution is to increase the number of file descriptors to unlimited.

By default, Linux only allows 1024 connections.

The presence of the agent can create the requirement for many more connections than usual.

The fix is to set "ulimit -n" in the shell that is running the application server (such as the run.sh).

"ulimit -n" sets unlimited connections to the Linux/JBoss server, thereby allowed the agent and application to function properly.

Here is how you need to do it.

vi /etc/security/limits.conf

add the following 2 lines to increase default 1024 limits:

pophca1 soft nofile 4096

pophca1 hard nofile 8192

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s