Error notification feature for enterprise deployments

Riva server can be configured to write notifications to the Windows system event logging system.  Customers can implement notification by email, pager or other warning mechanism using common 3rd party Windows Event Monitoring services.

This article will discuss:

• Window Service for Riva Sync Engine
• Category of Riva errors
• Error thresholds that trigger Event Notification
• How to enable and configure Riva Notification Features
  • Enable Riva Notification
  • Adjust Threshold Levels
  • Successive Notification Hour
• Email Notification in Sync Policy
  • Understanding Email Notification Emails

Windows Service for Riva Sync Engine

The sync engine can only raise notifications if the service responsible for the synchronization is up and running.  If the service is not started or in a disabled state, no notifications will be raised.

An external monitoring process should be used to monitor and ensure that the Windows Service responsible for the synchronization is started.  If the service is not running, this external monitoring process should notify the appropriate teams.

The service name can be found by using the Windows Service Control Manager (services.msc) as illustrated by the screenshot below,

In some configurations there can be many synchronization services all running concurrently.  Ensure that all services are monitored.

Category of Riva Errors

Notifications for catastrophic, critical and connection errors are raised by the application framework.  Error notifications will be raised to the standard “Windows Event” systems of the local Riva server.

There are 5 categories of errors that can occur.  Each priority correlates to a different severity:

• Critical Errors – Severe errors that prevent the sync from continuing. 
  Note: The most severe conditions must be resolved immediately.  Example:  Approaching or out of disk space.

• Connection Errors – Errors establishing a connection to CRM or Exchange. 
  Note: These include permission configuration issues.

• Cycle Errors – Errors that result in the sync cycle being ended prematurely. 
  Note: Indicates a specific user is no longer synchronizing as expected.

• Module Errors – Errors that result in a single module ending prematurely within a cycle.   
  Example:  For a specific user, the contact module completes, but the appointment module fails.

• Item Errors – Errors synchronizing a single item.
  Example: A CRM validation rule does not allow a specific value.

Riva Error Thresholds that Trigger Event Notification

Often some types of errors can resolve themselves, such as network connectivity being restored.  A number of thresholds are in place to allow for such recovery without raising immediate notification:

• Critical Errors – Notification raised as soon as they are encountered – no threshold.

• Connection Errors – Notification raised if the error is encountered 5 times in a row.

• Cycle Errors – Notification raised if the error is encountered 5 times in a row.

• Module Errors – Notification raised if the error is encountered 5 times in a row.

• Item Errors – Notifications are not raised for individual item errors as the reasons for these are numerous and extremely varied. Individual item error notification would create too much “noise” and make it difficult to evaluate when action needs to be taken.

A particular notification will only be raised once every 24-hours.  If an error continues to occur and a notification has already been raised, the next notification will be delayed for 24-hours or until 8 AM the next day, whichever is earlier.

How to Enable and Configure Riva Notification Features

Minimum Riva Version - Riva release 2.4.30 or higher must be installed.  Upgrade Riva to the latest release if required.

Enabling Riva Notification Features - is enabled and configured by applying a set of advanced option keys and values in the applicable Omni CRM Agent Service .CONFIG file:

• Riva for Exchange - Omni.Riva.CrmAgentEx.exe.config
• Riva for GroupWise - Omni.Riva.CrmAgent.exe.config

Service Restart Required - Once the advanced option settings are saved to the .CONFIG file, the Omni CRM Agent service needs to be restarted before Riva Notification Features are applied.

Enable Riva Notification

Apply the following advanced option key and value to enable or disable Riva Notification to Windows Event Logs:

Sync.Crm.Notifier.Enabled = true / false (true enables Riva Notification, false disables Riva Notification)

Once enabled, Riva will create a "Riva" Event Log and write notifications to that Windows log.

Adjust Threshold Levels (Optional)

The default threshold levels for connection, cycle, and module errors is 5 errors repeated in a row.  This threashold can be set to a different value.  Apply the following advanced option keys and values:

Sync.Crm.Notifier.ConnectionErrorThreshold = 5 (set the desired integer value)
Sync.Crm.Notifier.CycleErrorThreshold = 5 (set the desired integer value)
Sync.Crm.Notifier.ModuleErrorThreshold = 5 (set the desired integer value)

Successive Notifications (Optional)

By default, if Riva notification errors continue past the threshold levels, successive notifications are sent to the Windows Event Logs in 24 hours or at 8 am (local time), whichever is earlier.  This notification time can be set to a different hour:

Sync.Crm.Notifier.ResetTimeHour = #  (Provide an integer between 0 and 23 to specify an hour on a 24 hour clock)

Email Notification in Sync Policy (release 2.4.36+)

Riva 2.4.36+ includes an "Email Notification" page in the Exchange and Notes sync policy that provides the ability to configure email notification.  Refer to Email Notification in Sync Policy for instructions.

Understanding Notification Emails

Once email notification is enabled, Riva will send notification emails to the individual email addresses defined in the Email Notification page of the sync policy.  For a Delete Safety error similar to:

Once the default threshold of five identical consecutive errors is reached, Riva will send a notification email similar to:

Refer to the following information to properly interpret these types of emails:

• The line below the Subject provides a plain language description of the error.
• The "User" field identifies the user that Riva was syncing when the notification was tripped.
• The "Reference ID" field identifies the thread number recorded in the log, for example: EKNJUXLEPB53
• The "Thread" identifies a name identifier for the thread that is recorded in the log, for example: CTP-hta #895
• The "Type" field is a short name for "Event Type", which can be a number of different values based on which component was raising or is affected by the event itself - keeping in mind that not all events are errors.
  • Agent
  • Policy
  • Cycle
  • Folder
  • Item
  • Module
• The "Subtype", this is a cascading value based on the "Event Type", so an example might be when Riva starts, an Event Type of "Agent" is raised with a subtype of "Agent Starting". And when the system is shutdown, an event is raised with an Event Type of "Agent" with a subtype of "Agent Stopping".  "Subtype" can include:
  • Update
  • Create
  • Delete
  • Error
  • ItemSkipped
  • CompletedType
  • StartedType
  • StartSync
  • EndSync
  • ConnectionError
  • ConnectionAttempt
  • ConnectionEstablished
  • ItemMoved
  • Initializing
  • AgentStarting
  • AgentStopping
  • AgentRunning
  • Enqueuing
  • TimerReset
  • ItemMatched
  • PolicyStopping
  • PolicyStopped
  • PolicyStarting
  • CriticalError
  • CatastrophicError

For the "Error Notification" feature, it's wired into the error system, so only when errors occur, meaning you'll mainly have a subtype of:

• Error
• ConnectionError
• CriticalError
• CatastrophicError