Error: “java.lang.OutOfMemoryError: GC overhead limit exceeded” causing a Diagnostics Server crash

When running HP Diagnostics Server (various versions) with the Sun JVM 1.6.0_18, the Diagnostics Server is found to crash frequently due to a memory leak with messages of the form:

2011-04-05 16:07:52,582: SEVERE common : Unexpected Error [java.lang.OutOfMemoryError: GC overhead limit exceeded] occurred in thread [Probe Puller Main S_PULLER_IDLE_WAIT]

java.lang.OutOfMemoryError: GC overhead limit exceeded

and

2011-04-05 16:39:45,706: SEVERE mediator : Out Of Memory on thread Thread[Probe Puller Task,5, HP Diagnostics Server]: java.lang.OutOfMemoryError: GC overhead limit exceeded

and

2011-03-23 15:49:06,447: SEVERE equentEventScheduler: While running rare event: Calculate availability for Mercury System/Mercury System/<servername>

java.lang.OutOfMemoryError: GC overhead limit exceeded

being found in the "server.log" file. A JVM crash dump file may also be present indicated an issue with the garbage collection functionality for example it may contain:

Current thread (0x0000000000479800): GCTaskThread [stack: 0x0000000005f50000,0x0000000006050000] [id=1200]

or

Current thread (0x67f80400): GCTaskThread [stack: 0x680d0000,0x68120000] [id=2064]

This issue is due to a bug in the default garbage collector chosen by the JVM. For more information on this issue see:

http://forums.oracle.com/forums/thread.jspa?threadID=1539972&tstart=165

A workaround for this issue is to specify that the liner garbage collector should be used by the JVM by adding the "-XX:+UseSerialGC" flag to the server.nanny file located at:

<diagnostics_server_install_dir>\nanny\windows\dat\nanny\

for Windows. To add this flag modify the start_<platform> line in this file which should then look like:

start_nt="C:\MercuryDiagnostics\Server\jre\bin\javaw.exe" -server -Xmx512m -XX:+UseSerialGC

-Dsun.net.client.defaultReadTimeout=70000

-Dsun.net.client.defaultConnectTimeout=30000

"-javaagent:C:\MercuryDiagnostics\Server\probe\lib\probeagent.jar"

-classpath "C:\MercuryDiagnostics\Server\lib\mediator.jar;

C:\MercuryDiagnostics\Server\lib\loading.jar;

C:\MercuryDiagnostics\Server\lib\common.jar;

C:\MercuryDiagnostics\Server\lib\mercury_picocontainer-1.1.jar"

com.mercury.opal.mediator.util.DiagnosticsServer

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s