How to initiate a Contact Expert incident investigation

In case the deployed Contact Expert system is showing unexpected, erroneous behaviour, the first few steps are crucial for a successful root cause analysis regardless of whether the customer subscribed to any Geomant maintenance plan.

Check Contact Expert and Environment Configuration

The very first thing to do is to check whether the problem might have been caused by Contact Expert configuration settings.

Were the campaigns set up properly and are they in a running state?
Were the imported contact lists checked properly to filter out typos, phone number format differences, etc?
Are the CE dialing rules properly configured with how the alternate phone numbers are filled out in the imported contact list? Does CE call the alternate numbers you expect?
Were the After-Call-Work and WrapUp time periods set to an appropriate value so agents have time to perform call logging operations?
Were the agents observing the problems configured with the proper skill?
Were the scripts to be loaded to the agents observing the problems configured and created properly and are they showing the exact data they should?

Second, check whether the environment configuration is in line with CE prerequisites.

Telephony (Skype for Business, Avaya, etc) configurations
- Were all agents and CE campaign endpoints granted the proper Skype for Business policy permissions? (internal, external, international call permissions, etc.)
- Were all agents assigned to the Skype for Business pool the CE campaigns are also assigned?
- Are number normalization rules, PSTN routes, groups, queues in line with your expectations?
- Was there a recent failover between Skype for Business Front-ends in a high-availability Skype for Business topology? Is Skype for Business in a nominal, operational HA state at the moment?
- Is there a problem with Skype for Business's own database back-end?
CE database configurations
- Are the CE databases (ACESystem and ACEReport) accessible?
- Are the CE Administration and Supervisor sites accessible and can you load statistical information?
- Is the CE database in a responsive state, how fast do statistical information, agent side campaign scripts and customer contact history data show up? Are there long or varying length of wait times before the data is presented?
- Were the user login permissions for the above databases changed by a manual or automated process (i.e. ActiveDirectory group policy enforcements, etc.)
Media gateway or PSTN configurations
- Were the calls traversing through any media gateways before hitting Skype for Business and CE, and can the issue be observed in there?
- Were the erroneous calls treated in the gateway the exact way you would expect?
- Are the gateway's number normalization rules in line with your expectations?
Network
- Were all necessary components (gateway, Skype for Business servers, CE servers, database servers, back office servers if integrated and client desktop computers) able to communicate at the time the incident happened on all ports defined in the respective prerequisites?
- Was there any packet prioritization established for voice data on the network segments between te above components at the time of the incident? Did that work properly to sufficiently prioritize and/or segment the voice data traffic?

Rule out human error

If none of the environmental factors could be at play for the observed problems, the next most frequent root cause category is when users' assumptions on how certain Contact Expert features work are not always in line with how they actually work. Double-check all relevant Contact Expert and environmental configuration that could realistically be related to the problems observed.

Save problem evidences

In case the environmental and human factors were ruled out, all Contact Expert logs and debug information should be collected.

Collect Contact Expert server side logs

Enable server side logging

Log files containing detailed operational traces are the single most frequently used tools during issue resolution work. Allowing them to grow big on a production system risks jeopardizing available storage space so a balanced logging configuration must be achieved that would retain as old information as possible while minimizing storage space usage.

To review the CE Server Side logging configuration:

Log on to the CE Core Server computer(s).
Open the CE Tools shortcut and launch the CE PowerShell tool.
Execute the Get-CELogDetails cmdlet. This will list the name, file location, max file size and max number of log file generations to keep. The cmdlet displays the data in table format – if a column is partially visible, you can add extra format commands with the usual PowerShell piping method. For example to only list the log file paths and thus achieve wider display, issue this command:
```
Get-CELogDetails | Format-Table -Property LogFile
```

To change the logging settings of all CE components in one run:

Log on to the CE Core Server computer(s).
Open the CE Tools shortcut and launch the CE PowerShell tool.
Execute the Set-CELogDetails command with the appropriate parameters. You can get further help with the parameter config by issuing the Get-Help Set-CELogDetails command.

Collect server side logs

Log on to the CE Core server and find the folder the application was installed into. The default CE server side installation path is C:\Geomant\CE
Open the Logs folder and check that no .ZIP archives are present from an older investigation. If you find any old .ZIP or duplication of old log files, delete them.
Go back to the CE install folder (one level up from Logs), right click the Logs folder and select the Compress option to zip the whole directory with all files within. Make sure to name the compressed archive file in a unique manner that designates the source of the logs (i.e. 'services') as well as the current date and time to the minute.
Find and compress the web application component logs the same way. The default path for these are:

C:\Geomant\CE\Tomcat\webapps\ClientAccessServer\Log\*

C:\Geomant\CE\Tomcat\webapps\ACEEmail\Log\*
Repeat these steps for each individual CE server in your deployment.
Copy the generated zip files to a temporary location where you frequently bump into them and can easily remove them when they are not needed anymore.

Note

Depending on log settings and contact load, a Contact Expert system can generate fairly large number of large sized log files. Keeping the log detail level high forces the system to spend a lot of time and resources actually writing all the details into these files – time and resources that could otherwise be used to do business work. Also, keeping many log file copies can seriously jeopardize available free space and thus can also adversely affect the system operation.

Collect client side logs and evidences

If the symptom of the observed incident appears to affect the Contact Expert client, then collect the client side logs as well. The log file is stored at the following location by default: C:\Geomant\CE\Servers\AgentClientApp.Net\log

However it is possible to change this path in the CE Agent Application's config file – which can be found at the default location: C:\Geomant\CE\Servers\AgentClientApp.Net\AgentClientApp.Config.XML

This HowTo article describes these configuration parameters in detail -- you will need to look at the GC_LOG_DIRECTORY parameter's ignore tag to confirm it is set to no. If it is set to yes then no log file is printed to the path defined in the value tag.

If required, you can also enable printing log output into the Windows Event Logs that can be visited using the Event Viewer admin tool. For this click the Preferences tab on the ribbon and check the Create Windows Log option. Try to reproduce the error or wait for it to occur again to have the log accumulate any evidences.

Report the incident

Would you have a maintenance subscription plan with Geomant or a certified partner, you can now report the incident on their helpdesk. Make sure to explain the symptoms of the issue as detailed as possible. As a minimum the following piece of information are essential, missing any of these will render further investigation proportionally difficult or downright impossible:

Exact date and time of the incidents that the support staff should investigate in the logs provided
Call details like caller and called numbers, agent IDs, campaign names, email details, chat details
Collected logs
Any screenshots that help visualize the problem symptoms