Possible causes of a Knowledge Management Index crash

There are a variety of reasons why a knowledge document can terminate an index. It is not always black and white and that each needs to be reviewed on a case by case basis. Below lists some of the common causes of indexing issues. Keep in mind that 99.9% of all problems with indexing are with ServiceCenter / Service Manager and not the search engine. The version of Service Manager being used will determine which Search Engine is being utilized by Knowledge Management. Service Manager 9.2x and earlier will use the Verity K2 Search Engine. Service Manager 9.30 and newer uses the SOLR Search Engine.

A. Attachments

Probably the number one cause is attachments. Out of the box HP excludes a small set of file extensions: jpg;bmp;gif;exe;unl. This means that when Service Manager parses the knowledge document it will ignore those attachments that contain one of those file extensions. This list is rather small and should be expanded to include: png;psd;tif,js;jsp;dll. There are probably others, but these have been the most common.

If  additional extensions are not listed it does not necessarily mean the index will fail. In some cases Service Manager will parse the record and submit the document to the search engine, but the search engine rejects the document and does not index it. In earlier versions of ServiceCenter/Service Manager (i.e. 6.2.x through 9.2.x) the rejection by the search engine was something we had to actively search for by reviewing the <search engine>\\data\\services\\KM-INDEX\\log\\status.log. The errors in this log would give direction on how to approach a document that was rejected by the search engine.

Items to check when configuring to support attachments:

1. What types of attachments are most likely to be attached to a knowledge document?

2. How large is the attachment? If a large Word document then perhaps the External document type should be used so the actual Word document becomes the knowledge document instead of an attachment with a link .

Here is a list of the extensions that will be indexed by the Verity K2 Search Engine: .pdf, .rtf, .mif, .wpd, .htm, .html, .shtml, .asp, .cgi, .php, .sml, .xml, .txt, .text, .c, .h, .cpp, .cxx, .pl, .mbx, .doc, .xls, .ppt, .aw, .eml. Anything not on this list will – more than likely – be rejected by the search engine even if Service Manager parses the contents of the attachment and transmits that data to the indexer. Additionally, password protected files will not be indexed by the search engine.

NOTE: There is currently a 40 character limit at the database level for the “Skip these extensions” field (skipexts in the database). When you combine the out of the box extensions with the expanded list of extensions recommended in this article, the character count exceeds 40. Reindexing of the knowledge libraries may fail due to the system truncating what is included in that field.

The SOLR Search Engine uses the Tika parser. To see what types of files can be indexed by SOLR simply review the list within Service Manager called: “kmmimetypes”

B. Documents with invalid characters

ServiceCenter / Service Manager and both the Verity K2  and SOLR Search Engines accept standard html and xml tags. Special characters need to be wrapped via standard html. In many cases proprietary tags will cause problems. The most common tags are the Microsoft Office ‘mso’ tag. These tags can be inserted when a user copies and pastes from an application like Microsoft Word. What SHOULD happen, when a copy and paste method is used, is that a cleanup window (which is called the “Tidy” routine) should pop up notifying the user that they should paste the contents of their document into the Tidy window so the proprietary tags are cleaned up and replaced with standard tags. This should happen for anything that contains non standard formatting tags.

Non-standard formatting tags will not cause the index to fail, but what will usually happen is that all data up to the point of the special character is indexed. Everything after that special character is not indexed. So, if the term being searched on happens to be after that special character that specific document will not list in the hitlist.

Determining this can be complex and if you feel you are running across such a scenario then the first thing to try is to send the document in question to the workflow, edit the document and then click on Source button so you can see what the special character is wrapped in. If you see MSO anywhere then this is part of the problem with this document.

C. Knowledge Documents with LOTS of data or massive tables (This applies only to systems using Verity K2 Search Engine)

Documents, incidents and interactions will fail when there is more than 32k of data contained within a single field. In most cases the index will fail with a Signal 11 or flat out kill the client. This problem DOES NOT occur beginning with Service Manager 7.11 because the KMAdmin script has been updated to take into account larger data. 

D. File locking bug (This applies only to systems using Verity K2 Search Engine)

There is an issue with KMUpdate locking the file to be indexed in versions 6.2.1 through 6.2.5 and 7.0.1 and 7.0.2. When KMUpdate wakes up to index the kmknowledgebaseupdate records for a specific file it must lock that file. If an admin is reviewing that specific library via Manage Knowledgebases at the same time KMUpdate SHOULD NOT index that Library, but it does and thus will never close the index.

What should happen is that if KMUpdate is running the KMAdmin should not be able to perform any function on that library and vice-a-versa. This is resolved with ServiceCenter 6.2.6+ with the 6.2.6 KMUnloads and in Service Manage 7.10+.

Once this problem is experienced the only method of clearing the problem is to stop and restart both the search engine and the ServiceCenter server application.

Overall most everything will index without issue. As mentioned above many index issues need to be examined on a case-by-case basis.

How to install and configure Smart Analytics 9.41 in Server Manager

This article explains how to install and configure Smart Analytics 9.41 in Service Manager.

1. System
– Windows 2008 R2 SP1
. hostname: e616
. ip :
. Service Manager
. Smart Analytics

2. References
– Service manager 9.41 Smart Analytics Administrator and User Guide
 – Service Manager 9.41 – SmartAnalytics_Windows (installable bits)
 – Service Manager 9.41 Applications Patch Manager Guide

3. Source
– sm9.40.0015 app
– sm9.40.0015 rte
– sm9.40.0015 web
– sm9.40.0015 km

4. Target
– sm9.41.0020 app
– sm9.41.3016 rte
– sm9.41.3016 web
– sm9.41 Smart Analytics 

 5. Install sm9.41.0020 rte/eclipse
– install sm9.41.0020 rte
– install sm9.41.0020 eclipse
– in sm.ini, comment it out because it is not recognized in 9.41 // KM works even though commenting it out
– load platform_unload

6. Upgrade to sm9.41.0020 app
– reference : Service Manager 9.41 Applications Patch Manager Guide
 – disable client side unload at eclipse
– sm.ini
– System navigator > System Administration> Ongoing Maintenance > Patch Release
-> click Apply Patch Release
-> pick up file : D:\9.41\sm9.41.0020_Applications_HPSM_00743\sm9.41.0020_Application\application-9.41.0020-patchrel\AppPatchMgr\SM9.40-9.41.0020\patchrel.unl
-> current : 9.40.0015
target  : 9.41.0020
–> took 20 min
You need to manually map the following CI related blob fields to an alia table,
otherwise users cannot search by this field and the system will display an
error message:
cm3r: affected.services
cm3t: asset
probsummary: affected.services
requestTask: CIListCMDB,CIListContext
rootcause: affected.ci
For detailed steps, refer to the Overview of the Logical Name Solution and Its
Implications white paper (KM01653844).

–> in author’s verification, rootcause: affected.ci –> blob , the others are already CLOB.

– review results
System navigator > System Administration> Ongoing Maintenance > Patch Release
-> click View/Merge Results
type & action plan
Added (2709): no further action is necessary
Already Current (30) : no further action is necessary
Error (0):
Forced (0): no further aciton is necessary
Merged :
Renamed (16) : Keep the old version / Keep the new version / Merge new and old version ( 5 )
-> merge(copy all from left to right) -> mark as reconciled
ScriptLibrary, KMDocPreview , APM
ScriptLibrary, KMSearch, APM
displayscreen, kmdocFlow.viewENG, APM
format, kmknowledgebase.search.g.mobileen, APM
kmdoctype, errormsg, APM
kmdoctype, external, APM
kmdoctype, howto, APM
kmdoctype, probsol, APM
kmdoctype, reference, APM
kmknowledgebase, Incident_Library, APM
kmknowledgebase, Interaction_Library, APM
kmknowledgebase, Knowledge_Library, APM
kmknowledgebase, KnownError_Library, APM
kmknowledgebase, Problem_Library, APM
menu, ESSSM-Approval-Mobile, APM
menu, ESSSM-Mobile, APM

Previously Reconciled (0) : Keep the old version / Keep the new version / Merge new and old version
Updated: no further aciton is necessary

 – apply platform unloads
> no need because sm9.41.0020app has it.
for example, Author verified taht QCCR1E124742_SM941_SM930.unl is same to sm9.41.0020app.

7. Install sm9.41.P3 eclise/RTE
# by Author’s testing,
– after upgrading to sm9.41.0020 app, its KM does not work due to some defect of KM script library(9.41.0020 app).
but it is fixed by 9.41.3009p3 KM patch,  so, Author is going to sm9.41.3009 p3 rte/km/web/src/mobile directly.
– error message without QCCR1E128475_SM941P3_SM940.unl ( included in 9.41.3016p3 KM )
-> Script ‘KMCollection_Update’  line 658:
ERROR TypeError: lib.KMUtils.getSecurityFields is not a function at char 1
–> this function is not here in this KMUtils

– install sm9.41.3016p3 eclipse
– install sm9.41.3016p3 rte
load platform unloads,

8. Install sm9.41.3016p3 KM
– stop KM search engine
overwrite files,
from : D:\sm9.41.0020_Knowledge Management_HPSM_00744\sm9.41.0020_KM\knowledgemanagement
to   : C:\Program Files\HP\Service Manager 9.40\SearchEngine

 – load QCCR1E128475_SM941P3_SM940.unl

 – since 9.41, KM can use solr engine or IDOL search engine.
# when using SOLR engine
– install jdk 1.8(update 51 or greater)
– set system variables
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_60
– start KM

– sm.ini
– start SM server

 – rebuild index
-> in order to speed up, execute custom script:

– verify KM searching

 9. Install sm9.41.3016p3 web, sm9.41.0039p3 SRC, sm9.41.3000p3 Mobile
– install sm9.41.3016p3 web
. deploy sm9.41.3016p3.war
. modify web.xml

– install sm9.41.0039p3 SRC
. load unload files
. deploy src-9.41.0039p3.war
. modify applicationContext.properties

– install sm9.41.3000p3 Mobile
. load unload files
. deploy webapp-9.41.3000p3.war

– verify connection

10. Install smart analytics 9.41
– execute, setupSmartAnalyticsWindowsX64.exe
– folder : D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics
– Choose Installation type : Quick install
– Configure SM and Local Machine IP
SM Server IP    :
Local Machine IP:
– Configure Client IP
Content Server IP   :
Content Server IPv6 : fe80::c507:f93:2e54:f57e%10
// IPv6 should be set because Smart Analytics uses it internally, page 13 of Service manager 9.41 Smart Analytics Administrator and User Guide
// how to verify IPv6 address :
//   > ping fe80::c507:f93:2e54:f57e%10

 – SM Smart Analytics Server
* HP SM Smart Analytics Server
Server  Port : 9000
Index   Port : 9001
Service Port : 9002
* Main Content Server
Server Port  : 10010
Index  Port  : 10011
Service Port : 10012
* Smart Search Proxy Server
Service Port : 20010
Replicas(0 or 1): 0
Index Port   : 20011
Service Port : 20012
* Smart Search Content Server1
Server Port  : 30010
Index  Port  : 30011
Service Port : 30012
* Smart Search Content Server2
Server Port  : 30020
Index Port   : 30021
Service Port : 30022
* Image Server
Server Port  : 18000
Service Port : 18001
* Connect Framework Server(CFS)
CFS Port     : 7000
CFS Service Port: 7001
-> installing Smart Analytics, Visual C++ 2010 x64,
-> start service  (memory usage: 2.62G -> 4.42G )
-> registered services
HP SM Smart Analytics CFS Connector
HP SM Smart Analytics Content1
HP SM Smart Analytics Content2
HP SM Smart Analytics Content3
HP SM Smart Analytics Image Server1
HP SM Smart Analytics Main Server
HP SM Smart Analytics Smart Search Proxy

– configuration files
CFS       :                   D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\CFS\CFS.cfg
Content1  :                   D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content1\Content1.cfg
Content2  :                   D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content2\Content2.cfg
Content3  :                   D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content3\Content3.cfg
Image Server1:                D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\ImageServer1\ImageServer1.cfg
Main Server:                  D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\IDOL\AutonomyIDOLServer.cfg
Smart Search Proxy:           ../ImageProxyServer/dah.cfg
SharePoint  connector:    <Smart Analytics Installation>/SharepointRemoteConnector/SharepointRemoteConnector.cfg
OMNI  group  server:      <Smart Analytics Installation>/OmniGroupServer/OmniGroupServer.cfg
HTTP  connector:          <Smart Analytics Installation>/HTTPConnector/httpconnector.cfg
File system  connector:   <Smart Analytic Installation>/FileSystemConnector/filesystemconnector.cfg

 – apply Smart Analytic license on LicFile.txt (C:\Program Files (x86)\HP\Service Manager 9.40\Server\RUN\LicFile.txt)

– still SM9.41’s KM works with Solr
( We can not use Solr’s KM since enabling Smart Analytics.
Smart Analytics’s KM will be used instead of Solr when enabling Smart Analytics. )

11. configure Smart Analytics 9.41 including administrative task
# only IPv4 is supported for smart analytics with Service Manager

1) enable Smart Analytics in Service Manager
– System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Configuration
-> click “Enable Smart Analytics”
–> Questions window
“Once migrate to IDOL, you cannot use SOLR as search engine any longer and
it requires logoff. Are you sure to continue?”
—> click “Yes”  –> auto log off
– now, we can not use “SOLR” and there is no “Search Knowledge” munu in Knowelege Management.
– goto, System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Configuration
-> set URL for each services
* Smart Analytics Server:  -> click ‘Test Connection’
* Default CFS Server:  -> click ‘Test Connection’
* Image Server:  -> click ‘Test Connection’
-> click ‘Save’

2) configure data cleansing
– System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Configuration
> it is based on purpose

 3) configure Smart Ticket
– System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Smart Ticket
System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Configuration > click ‘Smart Ticket’ icon
> perform training
> perform testing ( after done of traing )
– verification
System Navigator > Service Desk > Create Smart Interaction
-> description : outlook down email
-> click ‘smart classification’
-> category:         compliment
subcategory:      Employee Off-boarding
Area:         authorization error
Affected Service: E-mail / Webmail (North America)
– some topics in Service manager 9.41 Smart Analytics Administrator and User Guide
page 38, Apply a rule-based training
page 39, Perform tuning in the smart ticket definition
page 40, Configure Smart Ticket for multi-company

 4) configure Hot Topic Analytics
# in the out-of-box system, three hot topic analytics configuration records are provided,
problem/known Error
– how to use it ( it work only in web  ):
System Navigator > Service Desk > Hot Topic Analytics
– how to configure:
System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Hot Topic Analytics
> start Index
> Stop Index
> Refersh Status

5) configure Smart Search
– how to configure:
System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Smart Search
> select one of knowledgebse Names -> click ‘Full Reindex’
– Index Status is changed : “Offline” -> “Not Started” -> “Indexing” -> “Finished”
– “KMUpdate” bg Process should be running and schedule file should has it, name=”KMUpdate”
> Author’s comment : if Index status=”Not Started” for a long time, need to check ‘KMUpdate’ bg process and its schedule record.
– knowledgebse Names,     TYPE,   Display Name,    Interval, Index Status, Doc Count :
Catalog_library       sclib  Catalogs            1     Finished   113
Change_library        sclib   Changes             1         Finished   35
Contact_Library       sclib  Contacts            1         Finished   509
Department_Library    sclib  Departments         1         Finished   47
Device_Library        sclib  Configuration Items 1         Finished   1067
Incident_Library      sclib  Incidents           1         Finished   210
Interaction_Library   sclib  Interactions        1         Finished   137
Knowledge_Library     sclib  Knowledge Library   1         Finished   166
KnownError_Library    sclib  Known Errors        1         Finished   0
Location_Library      sclib  Locations           1         Finished   6
Problem_Library       sclib  Problems            1         Finished   2
Request_Library       sclib  Requests            1         Finished   11

– how to verify:
> web login
> click icon, Magnify(next to command box)
> search keyword “*”     -> then, we can see all
Configuration Items(1067)
Knowledge Library(167)
Known Errors(0)

6) configure and monitor connectors
– it is for enabling search actions among different data sources
– how to configure:
System Navigator > System Administration > Ongoing Maintenance > Smart Analytics > Smart Search
> click ‘Connector Configuration’

 7) add the “idol.assistant” capability word to the operator records
– add ‘idol.assistant’ for falcon, Aaron.Caffrey
– ess self-service users are able to submit smart request records after you enable SM Smart Analytics.
– in 9.41, Aaron.Caffrey user can use ‘submit a smart request’ without idol.assistant (Author’s testing)
-> it seems something has changed in 9.41
-> “idol.assistant is required to use smart analytics”, page 54 of Service manager 9.41 Smart Analytics Administrator and User Guide

8) how to use smart analytics assistant
– sm login > command line > saa
-> “smart analytics assitant” is launched
-> this is adminstrative tool

12. Verify user’s task with Smart Analytics
1) create a smart ticket in ESS
– login to ESS page by Aaron.Caffrey
– submit a smart request
– type issue details then submit
2) create a smart ticket in SRC
– login to SRC
3) create a smart ticket in mobility client
4) create a smart tieket in power user view
5) create a problem by using hot topic analytics
problem managment > hot topic analytics
6) access hot topic analytics from incident reports
7) use ‘Smart Search’ for each libraries

# releate files
– configuration files
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\IDOL\AutonomyIDOLServer.cfg      // Main Server:
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\CFS\CFS.cfg                      // CFS
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content1\Content1.cfg            // Content1
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content2\Content2.cfg            // Content2
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\Content3\Content3.cfg            // Content3
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\ImageServer1\ImageServer1.cfg    // Image Server1
..\ImageProxyServer\dah.cfg                                                                   // Smart Search Proxy
<Smart Analytics Installation>\SharepointRemoteConnector\SharepointRemoteConnector.cfg          // SharePoint  connector
<Smart Analytics Installation>\OmniGroupServer\OmniGroupServer.cfg                              // OMNI  group  server
<Smart Analytics Installation>\HTTPConnector\httpconnector.cfg                                  // HTTP  connector
<Smart Analytic Installation>\FileSystemConnector\filesystemconnector.cfg                       // File system  connector

– log files
D:\Program Files (x86)\HP\Service Manager 9.41\SmartAnalytics\IDOL\logs\

SRC does not start when src.km.enabled=false at applicationContext.properties

SRC does not start even with src.km.enabled=false because it looks SM is trying to retrieve data from KM configuration

If SRC is configured to use KM in the KM environment record but at applicationContext.properties that is set to false, SRC will still try to retrieve data and connectivity to KM. If KM Search is not running fine or network problem exist that can cause SRC to not start

Please uncheck SRC setting at KM environment record to allow SRC to start normally

How to fix error “Failed to save branding configuration” in SM webtier

While saving the branding configuration if you are getting the below error message, as shown in the screen shot.

image text

To perform any branding operations, it is required to enable the branding feature by specifying a location to store your branding files.

1. On the web tier, create a folder (e.g. C:/customize). This folder will be the root folder to store your branding image files.

2. Stop the web application server.

3. Specify the folder location in the customize-folder web parameter. In the following example, we add C:/customize to the parameter value field:


4. Restart the web application server.

In the web.xml, the customize-folder is not set properly

<param-value>C:\Program Files (x86)\HP\Service Manager 9.40\customize</param-value>

It should be:

<param-value>C:/Program Files (x86)/HP/Service Manager 9.40/customize</param-value>

Also you must ensure the write access to customize folder.

Backslash is used in the folder directory in Windows by default, but to be compatible with other platforms, we suggest using slash instead.

The subtype field ‘SM’ is not being set by UCMDB integration

Out of the box CI and relation push from Univeral Configuration Managment Database (i.e. UCMDB)  to Service Manager (i.e.SM) appears to be working. CI’s are created together with correct upstream and/or downstream relations. On closer examination it can be seen that the relations pushed to Service Manager have not filled in the SM subtype being the actual type in UCMDB as shown in the screenshots below:

image text

The below screenshot shows the common_relations.xslt file where in comments the out of the box mappings can be seen.
image text
The relationship Type has only these  possible values in SM, Logical or Physical. Currently there is no support for any of the SM Physical relation types in UCMDB when sent over.

The following modifications can change the behavior.

·         RelationshipType has only two possible values in SM, logical or physical. Since there is no support for any of the SM Physical relation types in UCMDB these had to be hard coded into Logical.
·         RelationshipSubType contains all the relationship types as mentioned in UCMDB. (composition, usage,…). Changed the out of the box mapping of RelationshipType to RelationshipSubType.

The following screenshot illustrates the values correctly mapped in SM.

image text

“Too many open files” error in ALM wrapper log

When ALM is deployed on Linux and the error “Too many open files” exists in the wrapper logs, this is an indicator the “open file” limit on the Unix file system has been exceeded.

This can cause issues where attachments can’t be opened, tests executed, inability to open scripts or save scripts.

By default the “open file” limit on Linux is set to 1024; the number of concurrently open files.

To determine the “open file” value type the following in the Unix shell…

utlimit -a

Many results will be returned. Examine the value of the “open files” parameter.

To correct do the following…

1) Using vi or similar method, open the file /etc/security/limits.conf

2) Add the following lines to the file…

root soft  nofile <value>
root hard  nofile <value>

For example…

root soft  nofile 2048
root hard  nofile 2048

The value should be increased until the “Too many open files” error goes away. Start by doubling the current value.

3) Save the file
4) Restart Linux

Suggest checking to make sure the changes were implemented by issuing: ulimit -a
Then check the “open files” limit to make the changes took place.

Cannot add HTTPS Performance Center URL in Application Lifecycle Management LAB Management

ALM and PCS were configured to work with HTTPS, when adding the HTTPS PCS URL, the lab management shows an error message that it could not connect to the PCS through HTTPS .

ALM and PCS certificates are installed in the correct keystone, and both URLS work.

Please review with the customer Information Technology (IT) team if there are any IT policy that automatically disable TLS 1.0 after a Server reboot.

If there is a policy on disabling  the TLS 1.0 in the PCS machine, it can result in the PC machine rejecting the ALM communication.

Once TLS 1.0 was enabled, PCS servers with HTTPS URL can be added to the ALM LAB management.