OnCall IVR Engine v3.3.4, part of OnCall IVR Suite v3.3.x

Setting Up IVR Engine High Availability

Introduction

OnCall IVR Engine provides multiple different high-availability configurations depending on the telephony infrastructure it is deployed into. In a Microsoft telephony environment for instance, it relies upon the Microsoft architecture's inherent DNS load-balancing capability at the trusted application pool level, whereby two or more instances of the trusted application can be deployed into a pool in a load-balanced fashion.

The following sections describe the installation steps needed to achieve a high-availability state of the OnCall IVR Engine.

The following article describes how OnCall IVR operates with Contact Expert in case of a failover: Behavior of Contact Expert and OnCall IVR in High-Availability cases

High Availability With Microsoft Telephony

Implementing OnCall IVR Engine High Availability for the Microsoft Telephony platform involves using the DNS load-balancing capability of this infrastructure (further details are available at the Microsoft development websites).

It requires the followings:

Installation of the OnCall IVR Engine software onto more than one server computer – please follow the IVR Engine Installation for the individual configuration file settings
Windows Identity Foundation Windows Server Feature deployed on the IVR Engine servers – on top of the requirements detailed in the IVR Engine Prerequisites document
SfB Core components deployed on the IVR Engine server computers
DNS configuration for the pool and computers
Trusted Application Pool, Computers, Applications and Endpoints configured in the Microsoft telephony infrastructure

Installing Windows Identity Foundation

Launch the Server Manager administration tool on each IVR Engine server host with a user account possessing local administration privileges.
Locate the Features installation section of the Add Roles and Features wizard.
Locate and select the Windows Identity Foundation 3.5 component from the list and click Install.
Wait for the component to be installed and exit from the wizard. You might need to restart the computer for the changes to take effect.
Perform these steps on every other IVR Engine server host.

Installing SfB (Lync) Core Components

Install the SfB (Lync) Core components on every IVR Engine server host using the SfB (Lync) Server install tool bootstrapper.exe from its installation media:
```
  bootstrapper.exe /BootstrapLocalMgmt /MinCache
```
Add the user account configured to run the Geomant IVR Endpoint Service Windows service to the RTC Component Local Group user group, as well as the local administrators group.
Perform these steps on every other IVR Engine server host.

Create the IVR Trusted Application DNS Load-balancing Configuration

Launch the DNS management administration tool on a computer joined to the AD domain where you logged in with a domain administration account.
Create a new *A record* in the DNS zone where the OnCall IVR as well as the Microsoft telephony infrastructure are located and enter the name of the IVR trusted application pool that will contain the individual IVR servers. This FQDN will be the common name of the entire IVR application pool.
Add the IP address of the first OnCall IVR Engine server computer to this record A.
Repeat steps 2 and 3 for every subsequent OnCall IVR Engine server computer that are going to participate in the high-availability IVR pool.

Important
Use exactly the same hostname for all A records – this is essential for the DNS load-balancing to work.
Make sure the reverse DNS PTR records are also created.

Expand the Microsoft Telephony Topology to Include the IVR Engine Servers

In order to distribute calls between the IVR Engine server computers, we set up the Trusted Application Pool, Application and individual menu Endpoints with **the FQDN of the entire IVR pool*, and not with any of the IVR Engine server computer's FQDN. The IVR pool DNS *A record is resolved to the IP address of one of the individual IVR Engine server in a round-robin fashion – this is the load-balancing method by which the calls are spread across the computers.

Set up a multi-computer trusted application pool for the IVR Engine servers

Log on to one of the Microsoft Skype for Business (Lync) server computers that have the SfB/Lync Server administration tools installed.
Use a login account possessing domain administrator and RTCUniversalServerAdmins permissions.
Launch Microsoft's Topology Builder application in elevated mode (right-click and select Run as Administrator).
Expand the topology tree until you see the Trusted application servers node for your telephony site and infrastructure.
Right-click the Trusted application servers node, select Add application pool... and proceed creating a Multiple computer pool with the two (or more) IVR Engine servers for which you created the DNS A records previously.
Publish the updated topology.

Create a trusted application for the IVR Engine servers in the pool

Log on to one of the Microsoft Skype for Business (Lync) server computers that have the SfB/Lync Server administration tools installed.
Use a login account possessing domain administrator and RTCUniversalServerAdmins permissions.
Launch the Skype For Business (Lync) Management Shell in elevated mode (right-click and select Run as Administrator).
Execute the following PowerShell cmdlet to create a trusted application within the application pool established earlier:
```
New-CsTrustedApplication –ApplicationId "IVR" –Port "6000" –TrustedApplicationPoolFqdn "[IVR Pool FQDN]" 
```
TrustedApplicationPoolFqdn: Make sure you enter the FQDN of the entire IVR pool and not the FQDN of the individual IVR Engine server computers.

There is only one Trusted Application created for the pool and it will work for all the IVR Engine server computers in the pool. This also means that the IVR Engine must be configured to work on the same TCP port defined in the Trusted Application.

Create the IVR trusted application endpoints

Log on to one of the Microsoft Skype for Business (Lync) server computers with an account possessing domain administrator and RTCUniversalServerAdmins permissions.
Launch the Skype For Business (Lync) Management Shell in elevated mode (right-click and select Run as Administrator).
Execute the following PowerShell cmdlet to create the individual menu endpoints in the IVR pool:
```
 New-CsTrustedApplicationEndpoint –ApplicationId "IVR" –TrustedApplicationPoolFqdn "[IVR Pool FQDN]" –SipAddress "sip:[SIP URI of the IVR menu]" –DisplayName "[name of the IVR menu]"
```
TrustedApplicationPoolFqdn: Make sure you enter the FQDN of the entire IVR pool and not the FQDN of the individual IVR Engine server computers.

SipAddress: the hostname part of this URI is an arbitrary string that should clearly identify which IVR menu it corresponds to.

DisplayName: this information will be available to all SfB/Lync users via their SfB/Lync Client, so make sure it describes that this endpoint is an IVR menu and make it reflect what service the menu provides.
Execute this cmdlet for every IVR menu.

Update the Microsoft Telephony System Topology

Execute the following cmdlets with the Skype for Business (Lync) Management Shell still open:
```
Enable-CSReplica
Start-Service Replica
Invoke-CSManagementStoreReplication
```
Execute the following cmdlet to reveal the CMS replication status:
```
Get-CSManagementStoreReplicationStatus
```
If the status is not up to date, reboot the IVR Engine server and repeat from the Invoke cmdlet in step 1 (line #3).
The following steps will take care of the necessary certificates. Proceed only if you have not created any such certificate yet. The first step here is to execute the following cmdlet to reveal any pending certificate requests – proceed to the next steps if there is none:
```
Get-CSCertificate -List
```
Execute the following cmdlet to create a new certificate request:
```
Get-CSCertificate -New -Type Default -CA [CA Server FQDN]\[CA Authority] -Verbose
```
The [CA Server FQDN] is the fully qualified domain name of the certificate authority service that issued the certificate and [CA Authority] is the actual authority.

Make note of the Thumbprint displayed in the output of this cmdlet.
Execute the following cmdlet to define the certificate requested above to the Default server role:

Error rendering macro 'code': Invalid value specified for parameter 'theme'
```
Set-CSCertificate -Type Default -Thumbprint [Thumbprint from the previous cmdlet]
```

High Availability In a Standalone Implementation

Achieving HA in a standalone implementation is also possible if the telephony infrastructure – including the SIP gateway/PBX – supports alternate media routing services. Due to the high number of available 3rd party SIP gateway and PBX services, it is impossible to provide detailed technical steps, we will nevertheless highlight the key tasks a successful HA setup requires in such a setup.

Install the OnCall IVR Engine – or at least its EndpointService – in multiple copies on multiple server computers. If required, you can multiply the NodeServices as well.
Establish the PBX and/or SIP gateway configuration of each individual IVR Engine EndpointService instances.
Set up a failover media routing service of this telephony infrastructure across these IVR Engine EndpointService instances. When also having multiple NodeServices, configure these to work with the designated EndpointService instances.