Category: Service Manager

Service Manager troubleshooting tips and tricks

RTE E GetPreference DOS attack detected! occurs frequently in logs

Abdul RahimLeave a comment

The error

RTE E GetPreference DOS attack detected! Session will be terminated.

occurs very frequent for one servlet only (horizontal scaled system).

Example:

sm.cfg: sm -httpPort:13091 -httpsPort:13092 -sslConnector:1 -ssl:0

The message occurs more than 4000 times within 5 hours:

RTE E GetPreference DOS attack detected! Session will be terminated.

Check when the servlet was started by searching for string Initializing ProtocolHandler :

6361(  6361) 03/01/2019 15:16:09 Initializing ProtocolHandler [“http-nio-13091”]
6361(  6361) 03/01/2019 15:16:09 Initializing ProtocolHandler [“http-nio-13092”]

Result:

6361(  6361) 03/01/2019 15:16:09 Failed to initialize end point associated with ProtocolHandler [“http-nio-13092”]
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:350)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:810)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:476)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:120)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:871)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:135)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:347)
at com.hp.ov.sm.tomcat.EmbeddedTomcat.main(EmbeddedTomcat.java:421)
6361(  6361) 03/01/2019 15:16:09 Failed to initialize connector [Connector[HTTP/1.1-13092]]
This error tells us that the httpPort:13092 did NOT successfully start. It means only the plain HTTP (httpPort) started successfully.
It means that any client attempting to connect will not be able to because you have enabled TLS/SSL and TSO in SM.

The Cause of this issue is because 

java.net.BindException: The address is already in use

This indicates that on a previous shutdown of this servlet, the port 13091 did not shutdown properly.

To find out the time of shutdown of a servlet, search for the string
Stopping ProtocolHandler

You might need to monitor the sm_<pid>_stdouterr.log files for additional clues.

The https port (in our example 13092) is not free and still bound to some process(es) at the time the servlet was started up.

Why does the “DOS attack” error come up?

When SM load balancer forwards a request to this servlet, it does so first to httpPort to exchange the GetPreference SOAP message, then it will automatically switch to httpsPort.
This switch over to https did not occur in 10 seconds as the httpsPort did not start at all!

To Fix this issue

 

Before you start a servlet you need to check that no processes are bound to the ports anymore.

1) Check the server (linux in our example) for any processes still bound to this https port.

2) Stop that processes.

3) Start the servlet

4) Check log file for message

Initializing ProtocolHandler

In our example this message should come up for both ports, http and https :

Initializing ProtocolHandler [“http-nio-13091”]
Initializing ProtocolHandler [“http-nio-13092”]

You should not see these messages:

Failed to initialize end point associated with ProtocolHandler [“http-nio-13092”]
java.net.BindException: Address already in use

For future monitoring:

Monitor the logs on startup to ensure no messages of below type occur to prevent this type of issue:

Failed to initialize end point associated with ProtocolHandler

If you see those types of error strings, you must immediately shut down this servlet,  check to make sure all bound ports are unbound, then start up the servlet again.

Marquee not displayed in IE

Abdul RahimLeave a comment
Marquee in format sc.manage.ToDo.g is displayed in windows client and in web client with Firefox. But IE does not display the marquee (banner) .

 

Steps to reproduce:

A) Prepare login form to add a simple marquee field

1. command fd
2. format sc.manage.ToDo.g
3. Design
4. add a field of type Marquee
5. Save the format

B) Test with different browsers

1. login to Web Client with Firefox – the marquee is displayed
2. login to Web Client with IE – the maruqee is not displayed

The problem is a setting in IE options:

IE Internet Options
Advanced
Multimedia
Play animations in webpages* = true
Restart browser before testing again.

This solution is described here:
https://stackoverflow.com/questions/10527528/marquee-is-not-showing-in-internet-explorer :

As several people have observed that the marquee text works on IE, I suspect that this is about a setting on the IE you are using. At least if I go to Internet settings, Advanced settings, under Multimedia there is a checkbox for allowing animations. Apparently marquee is counted as animation in this sense, since when I checked the checkbox off (it is on by default) and restarted IE, the marquee text is not there (not even as static text).

 

Worker machine error referring to “dockerd”

Abdul RahimLeave a comment

In general, the worker machines are working correctly, but they are presenting the following error
dockerd: time=”2019-01-29T22:29:32.451238492-05:00″ level=warning msg=”containerd:
unable  to save 008f4389030eea47e9b8a306cfca1f2fa3a45689336b967f02291aef8422ab42:f0e52fe2e76259219fe1c8d8831bc2503e0e4a94aa088a2e4dbd6941801bd055 starttime: read /proc/22054/stat: no such process”.

This problem will occur once a pod is restarted/delete/create and there were pending actions by dockerd script while the pod is terminating any pending processes will be terminated and if the actions completed prior the processes are killed to close all pending references could potentially try to complete an action into a process that no longer exist and will fail with the warning message. Since this is a warning and NOT an error message it’s important to monitor if pods are constantly restarting because restart should occur occasionally and not in a consistent basis.

Unable to turn off Adaptive Learning for Smart Search in Service Manager

Abdul RahimLeave a comment

Disabling Adaptive Learning in the Knowledge Management Environment Record does not prevent adaptive learning records from being created in the kmadaptive learning table. This problem only occurs when using Smart Search/Smart Analytics and IDOL for Knowledge Management. The problem does not occur when using the SOLR Search Engine.

There is a defect within the IDOLAssistantUseSolution ScriptLibrary record.

The following steps below will disable Adaptive Learning for Smart Search:

1. Login to Service Manager

2. Go to System Administration > Ongoing Maintenance > Environment Records > Knowledge Managment Environment Record

3. Uncheck (i.e. set to false) Use Adaptive Learning to Enhance Search Results

4. Save

5. Logout then back into Service Manager

6. Go to Database Manager and access the ScriptLibrary table

7. Inside ScriptLibrary access the following script: IDOLAssistantUseSolution

8. Find this section of code:

function useSolution(record, solution, library, reference, keyword) {
var mapping = funcs.filename(record);
logger.debug(‘applying solution ‘ + solution + ‘ onto ‘ + record);

KMSearch.useSolution(mapping, solution, record);

var database = library.kbname;
// XXX: reference has to encode twice …
reference = encodeURIComponent(reference);
updateRanking(database, reference);
IDOLGlobalSearchAdaptiveLearning.appendPhrase(library, solution, keyword);
return record;
}

9. Comment the following line: IDOLGlobalSearchAdaptiveLearning.appendPhrase(library, solution, keyword);

10. The function should now look like this:

function useSolution(record, solution, library, reference, keyword) {
var mapping = funcs.filename(record);
logger.debug(‘applying solution ‘ + solution + ‘ onto ‘ + record);

KMSearch.useSolution(mapping, solution, record);

var database = library.kbname;
// XXX: reference has to encode twice …
reference = encodeURIComponent(reference);
updateRanking(database, reference);
// IDOLGlobalSearchAdaptiveLearning.appendPhrase(library, solution, keyword);
return record;
}

11. Save, Compile

12. Adaptive Learning is now disabled for Smart Search

Load files into Service Manager from the OS command line

Abdul RahimLeave a comment

1. Copy the unload file (eg unload_file.unl) into the <SM INST>\RUN directory

2. Open the terminal and navigate to Go to the <SM INST>\RUN directory

3. Execute the command:

For Microsoft Windows: sm -bg file.load <unload_file.unl> NULL NULL winnt

For UNIX: sm -bg file.load <unload_file.unl> NULL NULL unix

Where <unload_file.unl> is the name of the file to be loaded

4. A message will indicate the load succeeded.

SM 9.40: SURVEY MODULE: Configure Survey for Mobility Client

Abdul RahimLeave a comment

1. In SurveyParameterMapping, there is not predefined mappingValue for Mob

So add one in global list (List Name: SurveyPredefinedParameters) {“{$EMAIL}”,”{$NAME}”,”{$LANGUAGE}”,”{$WEBURL}”,”{$ESSURL}”,”{$EXURL}”,”{$EXTSID}”,”{$SURVEYURL}”,”{$MOBURL}”}

{“Recipient Email”,”Recipient Name”,”Language”,”Web URL”,”ESS URL”,”External URL”,”External Survey ID”,”InternalSurvey URL”,”MOB URL”}

2. Then add Mob URL for Interaction module for the particular Connector ID. In this example adding the parameters for Internal Connector.

Add mobURL in Survey Connector (for interaction parameters) as highlighted in below snapshot.

3. ScriptLibrary: Survey_TaskProducer, add below line at Line: 117:

paramMap[“$MOBURL”] = url.mob;

4. ScriptLibrary: Survey_Utilities. Add below lines:

Line 134: var url = {“web”:””, “ess”:””, “mob”:””};

Line 172: url.mob = lib.urlCreator.getMobilityURLFromQuery(record_id);

5. System Information Definition:

Add a testing value for “mobility.url” field:

http://mob-url.google.com

6. Schedule the Survey Interation record: S00003

7. Enable the SMSurvey in SMIS:

8. Make sure the Repeat Interval of Survey_Service_Agent within a few minutes:

9. Make sure the Repeat Interval of SMSurvey1 within a few minutes:

10. Now close any one of interaction record:

11. Check the record generated in eventout table:

Now we can see that the mobility.url has been displayed in the email message.

Forms Designer ‘Output Conversion’ property work for Export To Text/Excel

Abdul RahimLeave a comment

In Forms Designer there is a Property named “Output Conversion”. When used it can change the display of data in the GUI (i.e. display just the date of a record instead of the full date/time). However, the parameter will not extend to exporting the data of the Service Manager record to Excel or Text.

This property is used by RAD subroutines that mask the display of data on an output field, or check and validate the entry of data into an input field. If specified, the routine is called at runtime to convert the contents of the field before it is displayed on the screen. This only works within the client and does not extend to the export of data feature.

Note that at the time of this writing Enhancement Request QCCR1E137853 exists within Development to look into extending the capabilities of this Property to the Export To Text/Excel feature