RTE E GetPreference DOS attack detected! occurs frequently in logs

The error

RTE E GetPreference DOS attack detected! Session will be terminated.

occurs very frequent for one servlet only (horizontal scaled system).

Example:

sm.cfg: sm -httpPort:13091 -httpsPort:13092 -sslConnector:1 -ssl:0

The message occurs more than 4000 times within 5 hours:

RTE E GetPreference DOS attack detected! Session will be terminated.

Check when the servlet was started by searching for string Initializing ProtocolHandler :

6361(  6361) 03/01/2019 15:16:09 Initializing ProtocolHandler [“http-nio-13091”]
6361(  6361) 03/01/2019 15:16:09 Initializing ProtocolHandler [“http-nio-13092”]

Result:

6361(  6361) 03/01/2019 15:16:09 Failed to initialize end point associated with ProtocolHandler [“http-nio-13092”]
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:350)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:810)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:476)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:120)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:871)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:135)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:347)
at com.hp.ov.sm.tomcat.EmbeddedTomcat.main(EmbeddedTomcat.java:421)
6361(  6361) 03/01/2019 15:16:09 Failed to initialize connector [Connector[HTTP/1.1-13092]]
This error tells us that the httpPort:13092 did NOT successfully start. It means only the plain HTTP (httpPort) started successfully.
It means that any client attempting to connect will not be able to because you have enabled TLS/SSL and TSO in SM.

The Cause of this issue is because 

java.net.BindException: The address is already in use

This indicates that on a previous shutdown of this servlet, the port 13091 did not shutdown properly.

To find out the time of shutdown of a servlet, search for the string
Stopping ProtocolHandler

You might need to monitor the sm_<pid>_stdouterr.log files for additional clues.

The https port (in our example 13092) is not free and still bound to some process(es) at the time the servlet was started up.

Why does the “DOS attack” error come up?

When SM load balancer forwards a request to this servlet, it does so first to httpPort to exchange the GetPreference SOAP message, then it will automatically switch to httpsPort.
This switch over to https did not occur in 10 seconds as the httpsPort did not start at all!

To Fix this issue

 

Before you start a servlet you need to check that no processes are bound to the ports anymore.

1) Check the server (linux in our example) for any processes still bound to this https port.

2) Stop that processes.

3) Start the servlet

4) Check log file for message

Initializing ProtocolHandler

In our example this message should come up for both ports, http and https :

Initializing ProtocolHandler [“http-nio-13091”]
Initializing ProtocolHandler [“http-nio-13092”]

You should not see these messages:

Failed to initialize end point associated with ProtocolHandler [“http-nio-13092”]
java.net.BindException: Address already in use

For future monitoring:

Monitor the logs on startup to ensure no messages of below type occur to prevent this type of issue:

Failed to initialize end point associated with ProtocolHandler

If you see those types of error strings, you must immediately shut down this servlet,  check to make sure all bound ports are unbound, then start up the servlet again.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s