Implement Riva Disaster Recovery Procedures

Riva On-Premise now offers the ability to deploy a passive-passive fail-over Riva server to facilitate recovering from a crash of the Windows system that hosts a Riva On-Premise primary server. The Riva disaster recovery (DR) strategy requires enabling the DR features, implementing a backup schedule schedule for the primary Riva server, and using the Fail-over Riva server if the primary Riva server fails.

Contents:

• What is Riva disaster recovery (DR):
  • Supported email platforms, and
  • Required Riva version;
• Riva DR scenarios and procedures:
  • Scenario 1 - Replicating a mirror snapshot of the primary Riva server,
  • Scenario 2 - Passive fail-over Riva server, and
  • Scenario 3 - Daily backup of the primary Riva server to an offsite data storage location;
• Common Riva DR procedures and guidelines:
  • Configure the primary Riva server for DR,
  • Recommended daily backup guidelines for Riva DR, and
  • Test the DR and data sync restoration procedures.

What is Riva Disaster Recovery

The disaster recovery (DR) feature can be enabled on the primary Riva server. Once activated, the Primary Riva server is backed up to preserve the Riva server installation files and the data sync transaction records. If the primary Riva server suffers a non-recoverable crash of the Windows system that hosts the Riva server, Riva sync can be restored from the backup.

The DR feature includes custom options keys to enable CrmID tracking. After the options have been applied, the Riva sync service appends the Riva CrmID to each Exchange item that Riva syncs with corresponding data in the CRM.

When a DR-enabled Riva polls Exchange (or Notes) and the CRM for data, Riva checks all updates (new items, modified items, and deletions) for the existence of the Riva CrmID value:

• If there is no Riva CrmID attribute value, the item is considered a new update, and Riva processes it.

• If the Riva CrmID attribute value is in place, Riva compares the last mod date of the item and the last transaction date in the Riva transaction metadata files and determines if an update needs to be synced. We refer to this process as remapping the transaction records.

Supported email platforms

Riva disaster recovery is supported for the following Riva On-Premise servers based on email platforms:

• Riva On-Premise for Exchange that uses Riva EWS connections. Riva MAPI connections and sync policies are not supported.
• Riva On-Premise for IBM Notes.

Required Riva version

Riva On-Premise release 2.4.38 or higher is required.

Riva Disaster Recovery Scenarios and Procedures

Three scenarios are available:

• Scenario 1 - Replicating a mirror snapshot of the primary Riva server virtual machine,
• Scenario 2 - Daily backup to a passive fail-over Riva server, and
• Scenario 3 - Daily backup of the primary Riva On-Premise server to offsite storage.

These scenarios can be combined if corporate disaster restoration policy requires a minimum of two parallel recovery options:

• Scenario 1 as the primary DR solution and Scenario 3 as the alternate.
• Scenario 2 as the primary DR solution and Scenario 3 as the alternate.

Scenario 1 - Replicating a mirror snapshot of the primary Riva server virtual machine to a different data centre

In this option, a hot mirror of the primary Riva server virtual machine is replicated to an alternate data center. The advantage of this solution is that a backup of the server can be created without stopping the Riva service; and the Riva server in the alternate data center can be started with minimum effort or time lost. 

To implement the hot mirror server scenario:

1. Enable DR options on the primary Riva server.
2. Replicate a mirror snapshot VM.
3. Restore Riva on the mirror snapshot VM.
4. Enable daily backups of Riva on the mirror snapshot VM.
5. Rebuild the primary Riva On-Premise server VM.

Step 1 of Scenario 1

To enable DR options on the primary Riva On-Premise server:

1. In the Riva Manager application, edit each sync policy to remove any inactive user that should not be synced by Riva again.

2. In the Riva CRM Service Monitor application, ensure that all users are syncing without serious errors. Resolve any errors. If you need assistance, contact the Riva Success Team.

3. Enable the DR options.

4. In the Riva CRM Service Monitor application, ensure that all users continue to sync without serious errors for at least two sync polls before replicating a mirror snapshot VM.

Step 2 of Scenario 1

To replicate a mirror snapshot VM:

• Use the applicable procedures to enable replicating a mirror snapshot of the primary Riva server virtual machine (VM) to a fail-over data center in a different physical geographic location.

  This creates a disabled (offline) passive fail-over Riva server VM that can be provisioned in the event of a failure of the primary Riva server VM.

  Important Note: It is critical to ensure that the replicated mirror snapshot VM is disabled and not running while the primary Riva server is operational.

Step 3 of Scenario 1

To restore Riva on the mirror snapshot VM if the primary Riva server VM stops functioning:

1. Ensure that the primary Riva server is not operational.

2. In the alternate data center, start the passive fail-over Riva server VM.

3. On the passive fail-over Riva server, start the Riva Manager application and the CRM Service Monitor application.

4. In the Riva Manager application, on the menu bar, select Setup. Run connection tests for the CRM connection and all email connections. If connections fail, resolve network connectivity. Close and restart the Riva Manager application, and retest connections.

5. After the connection tests pass, on the menu bar, select Policies. Enable the sync policy with the fewest target users.

6. In the CRM Service Monitor application, start the service, and monitor user sync to ensure that it is restored.

   Riva remaps previously synced data and handles new data items that do not have a corresponding crmID assigned to the mail item.

7. If significant data sync errors are observed, stop the service, and contact the Riva Success Team.

   If data sync appears to be normal for users, do as follows: In the Riva Manager application, on the menu bar, select Policies, and enable the remaining sync policies. Monitor data sync for those users.

Step 4 of Scenario 1

To enable daily backups of Riva on the mirror snapshot VM:

Note: The passive fail-over Riva server can serve as the primary Riva server and is already DR-enabled.

• Take immediate action to
  • Replicate a new mirror snapshot of the current operational Riva server virtual machine (VM) to a different data center in a different physical geographic location to create a new passive fail-over VM, or
  • Implement a daily backup of the Riva folder and files to an offsite storage service.

Step 5 of Scenario 1

To rebuild the primary Riva On-Premise server VM:

1. If the passive fail-over Riva server has been syncing users for more than 24 hours and the plan is to restore the primary Riva server in the original data center, replicate a new mirror snapshot of the current operational Riva server virtual machine (VM) to the original data center or a new replacement data center to create a new primary Riva server VM.

2. On the operating fail-over Riva server, disable the sync policies, and stop and enable the Riva sync service.

3. To restore Riva data sync on the new primary Riva server, follow the steps of To restore Riva on the mirror snapshot VM.

4. Schedule the daily replication of a mirror snapshot of the primary Riva server virtual machine (VM) to a fail-over data center.

Scenario 2 - Daily backup of the primary Riva server to a disabled passive fail-over Riva server 

In this option, we recommend a timed script that

1. Stops the Riva service.
2. Backs up the required files directly to the passive fail-over server, by overwriting the existing files.
3. Restarts the primary Riva service.

This activity would normally be scheduled during the Riva blackout period.

The advantages of this solution are that

• It is easy to facilitate and schedule.
• It is less costly than replicating a mirror to a different data center.
• It requires minimal time to restore Riva data sync operations.

To set up a daily backup of the primary Riva server to a disabled passive fail-over Riva server:

1. Enable DR options on the primary Riva Server.
2. Build the passive fail-over Riva server.
3. Schedule a daily backup of Riva data to the passive fail-over Riva server or to offsite storage.
4. Restore data sync on the passive fail-over Riva server.
5. Rebuild the primary Riva server.

Step 1 of Scenario 2

To enable DR options on the primary Riva server:

1. In the Riva Manager application, edit each sync policy to remove any inactive user that should not be synced by Riva again.

2. In the Riva CRM Service Monitor application, ensure that all users are syncing without serious errors. Resolve any errors. If you need assistance, contact the Riva Success Team.

3. Enable the DR options.

4. In the Riva CRM Service Monitor application, ensure that all users continue to sync without serious errors for at least two sync polls before building the passive fail-over Riva server.

Step 2 of Scenario 2

To build the passive fail-over Riva server in an alternate physical location or data center:

On the active primary Riva server, perform these steps.

Note: The changes will be copied to the passive DR Riva server.

1. Ensure that the DR features have been properly set and that Riva has synced all of the users at least once.

2. In the Riva CRM Service Monitor application, stop the service.

3. In the Riva Manager application, disable all sync policies, and close the Riva Manager application.

4. Create a RivaDR.ZIP file that contains the files and folders. The log folders are optional.
   
   

5. Copy the RivaDR.ZIP file to the Windows host of the passive DR Riva server.

6. In the Riva Manager application, enable the desired sync policies, and close the Riva Manager application.

7. In the Riva CRM Service Monitor application, start the service.

On the passive DR Riva server, perform these steps.

Note: The objective is to create a mirror copy of the primary Riva server and test connections. Do not install the Riva services.

1. Prepare a host Windows system (preferably in a different physical location from that of the primary Riva server) that meets the same system requirements as the primary Riva server.

   Important Note: It is critical to ensure that the DR server system requirements are configured to be identical to those of the primary Riva server.

2. Extract the RivaDR.ZIP file to install the Riva server files. Do not install the Riva services.

3. Create desktop shortcuts for the CRM Service Monitor application and the Riva Manager application.

4. Start the Riva Manager application.

5. On the menu bar, select Setup. Double-click the CRM connection, and confirm that the DR custom options exist. Perform a connection test.

6. On the menu bar, select Setup. Double-click each Exchange or Notes connection, and perform a connection test.

7. Open Windows Services, look for Omni services, and disable all Omni windows services.

Step 3 of Scenario 2

To schedule a daily backup of Riva data to the passive fail-over Riva server or to offsite storage:

• On the primary Riva server, implement one of the following daily backup options:
  • Option 1: Back up Riva files as a RIVA.ZIP or backup file set (if using backup software) to an offsite storage service.
  • Option 2: Back up Riva files as a RIVA.ZIP or backup file set (if using backup software) to a folder share on the passive fail-over Riva server.
  • Option 3: Back up Riva files directly to the passive fail-over Riva server, and overwrite the existing Riva files and folders.

Step 4 of Scenario 2

If you have to recover from a non-recoverable crash (loss of) the Windows system that hosts Riva, update the DR Riva server from the latest backup, install the services (if not installed), and enable sync.

To restore data sync on the passive fail-over Riva server:

Note: Steps 1 to 4 are not required if you used backup option 3.

1. If the passive Fail-over Riva server is hosted in a VM server, create a snapshot.

2. If the passive Fail-over Riva server virtual machine is off or paused, start the Windows server or VM that hosts the DR Riva server.

3. Download the latest primary Riva server backup file from offsite online data storage (if required).

4. Extract the latest primary Riva server backup file, and save and overwrite the existing Riva files and folders.

5. Start the Riva Manager application.

6. On the menu bar, select Setup. In the right pane, double-click the CRM connection, and confirm that the DR custom options have been set. Perform a connection test.

   Several CRM connections (for example, Microsoft Dynamics XRM Connection) depend on the Omni SDK Connection Agent service to be installed. Start the CRM connection wizard, install the service, and close the wizard.

7. On the menu bar, select Setup. Double-click each Exchange or Notes connection, and perform a connection test.

   The Notes connection depends on the Omni SDK Connection Agent service to be installed. Start the Notes connection wizard, install the service, and close the wizard.

8. On the menu bar, select Policies. Ensure that all sync policies are disabled. Edit a sync policy, and save the policy; no changes are required. Select Yes to install the Riva sync service.

   If Riva does not prompt to install the service, that means that the service is already installed. Compare the version of Riva against the version of the service installed. A re-install of the service may be required. For assistance, contact the Riva Success Team.

9. On the menu bar, select Policies. Enable the sync policy with the fewest target users.

10. In the CRM Service Monitor application, start the service, and monitor user sync to ensure that it is restored.

11. If significant data sync errors are observed, stop the service, and contact the Riva Success Team.

    If data sync appears to be normal for users, do as follows: In the Riva Manager application, on the menu bar, select Policies, and enable the remaining sync policies. Monitor data sync for those users.

Step 5 of Scenario 2

To rebuild the primary Riva server, you use a copy of the operational fail-over Riva server.

Perform these steps on the operational fail-over Riva Server:

1. In the Riva CRM Service Monitor application, stop the service.

2. In the Riva Manager application, disable all sync policies, and close the Riva Manager application.

3. Create a RIVA.ZIP file that contains the files and folders. The log folders are optional.
   
   

4. Copy the RIVA.ZIP file to the Windows host for the rebuilt primary Riva server.

Perform these steps on the rebuilt primary Riva server:

Note: The objective is to re-create the primary Riva server by using the RIVA.ZIP from the recently disabled fail-over Riva server. Do not install the Riva services.

1. Prepare a host Windows system (preferably in a different physical location from the primary Riva server) that meets the same system requirements as the primary Riva server.

   Important Note: It is critical to ensure that the DR server system requirements are configured to be identical to the primary Riva server system requirements.

2. Extract the RIVA.ZIP file to install the Riva server files. Do not install the Riva services.

3. Create desktop shortcuts for the CRM Service Monitor application and the Riva Manager application.

4. Start the Riva Manager application.

5. On the menu bar, select Setup. In the right pane, double-click the CRM connection, and confirm that the DR custom options have been set. Perform a connection test.

   Several CRM connections (for example, Microsoft Dynamics XRM Connection) depend on the Omni SDK Connection Agent service to be installed. Start the CRM connection wizard, install the service, and close the wizard.

6. On the menu bar, select Setup. For each Exchange or Notes connection in the right pane, double-click it, and perform a connection test.

   The Notes connection depends on the Omni SDK Connection Agent service to be installed. Start the Notes connection wizard, install the service, and close the wizard.

7. On the menu bar, select Policies, and ensure that all sync policies are disabled. Edit a sync policy, and save the policy; no changes are required. Select Yes to install the Riva sync service.

   If Riva does not prompt to install the service, that means that the service is already installed. Compare the version of Riva against the version of the service installed. A re-install of the service may be required. For assistance, contact the Riva Success Team.

8. On the menu bar, select Policies. Enable the sync policy with the fewest target users.

9. In the CRM Service Monitor application, start the service, and monitor user sync to ensure that it is restored.

10. If significant data sync errors are observed, stop the service, and contact the Riva Success Team.

    If data sync appears to be normal for users, do as follows: In the Riva Manager application, on the menu bar, select Policies, and enable the remaining sync policies. Monitor data sync for those users.

Scenario 3 - Daily Backup of the Primary Riva Server to an Offsite Data Storage Location

This option is recommended if resources are not available for Scenarios 1 or 2. This scenario can also be implemented as a secondary DR procedure to Scenario 1 or 2. In this option, we recommend a timed script that

1. Stops the Riva service.
2. Backs up the required files to an off-site location.
3. Restarts the Riva service.

This activity would normally be scheduled during the Riva blackout period. The advantages of this solution are that

• It is easy to facilitate.
• It offers the most flexibility to restore the Riva server to an alternate location once it has been established.

This option would require restoring the backup file set to an alternate Windows host and then starting the Riva server.

DR procedures for this scenario include:

1. Enable DR Options on the primary Riva Server.
2. Perform a daily backup of Riva data to offsite storage.
3. Restore Riva.

Step 1 of Scenario 3

To enable the disaster recovery option on the primary Riva server:

1. In the Riva Manager application, edit each sync policy to remove any inactive user that should not be synced by Riva again.

2. In the Riva CRM Service Monitor application, ensure that all users are syncing without serious errors. Resolve any errors. If you need assistance, contact the Riva Success Team.

3. Enable the DR options.

4. In the Riva CRM Service Monitor application, ensure that all users continue to sync without serious errors for at least two sync polls before proceeding with step 2.

Step 2 of Scenario 3

To perform a daily backup of Riva data to offsite storage:

• From the primary Riva server, schedule a daily backup as a RIVA.ZIP or backup file set (if using backup software) to an offsite storage location.

Step 3 of Scenario 3

To restore Riva data sync to a Windows host server in an alternate location:

1. Prepare a host Windows system (preferably in a different physical location from the primary Riva server) that meets the same system requirements as the primary Riva server.

   Important Note: It is critical to ensure that the DR server system requirements are configured to be identical to the primary Riva server system requirements.

2. Download the latest primary Riva server backup file from offsite online data storage (if required).

3. Extract the latest primary Riva server backup file, and save and overwrite the existing Riva files and folders.

4. Create desktop shortcuts for the Riva Manager application and the CRM Service Monitor for Exchange application.

5. Start the Riva Manager application.

6. On the menu bar, select Setup. In the right pane, double-click the CRM connection, and confirm that the DR custom options have been set. Perform a connection test.

   Several CRM connections (for example, Microsoft Dynamics XRM Connection) depend on the Omni SDK Connection Agent service to be installed. Start the CRM connection wizard, install the service, and close the wizard.

7. On the menu bar, select Setup. For each Exchange or Notes connection, double-click it in the right pane, and perform a connection test.

   The Notes connection depends on the Omni SDK Connection Agent service to be installed. Start the Notes connection wizard, install the service, and close the wizard.

8. On the menu bar, select Policies. Ensure that all sync policies are disabled. Edit a sync policy, and save the policy (no changes are required). Select Yes to install the Riva sync service.

   If Riva does not prompt to install the service, that means that the service is already installed. Compare the version of Riva against the version of the service installed. A re-install of the service may be required. For assistance, contact the Riva Success Team.

9. On the menu bar, select Policies. Enable the sync policy with the fewest target users.

10. In the CRM Service Monitor application, start the service, and monitor user sync to ensure that it is restored.

11. If significant data sync errors are observed, stop the service, and contact the Riva Success Team.

    If data sync appears to be normal for users, do as follows: In the Riva Manager application, on the menu bar, select Policies, and enable the remaining sync policies. Monitor data sync for those users.

12. Configure a daily backup of the Riva files from this server to the offsite storage location.

    This ensures that DR procedures are re-established.

Common Riva DR Procedures and Guidelines

These procedures and guidelines can be used in one or more of the DR scenarios identified above:

• Configure the primary Riva server for DR
• Recommended daily backup guidelines for Riva DR
• Test the DR and data sync restoration procedures

To configure the primary Riva server for disaster recovery:

1. Set the following DR custom options for the CRM connection(s):

   Sync.Crm.AddCrmIdCustomField = true
   Ex.AddCrmIdCustomFieldPostOp = true
   Sync.Ex.LoadCrmIdPluginField = true
   Sync.Crm.BlockCreatesOnItemsWithCrmIdPluginField = true

2. If the SyncForceUserExclusivity = True option key is set in the Omni.Riva.CrmAgentEx.exe.config file, modify that file so that it identifies the current primary active Riva server name (for example, "RivaFS1") and the passive DR Riva server name (for example, "RivaDR1") by adding the following additional key:
   
   <add key="Sync.Crm.ReplacementRivaInstanceMachineId.[Source Riva server name]" value="[Target Riva server name]" />
   
   For example:

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <appSettings>
       <add key="SyncForceUserExclusivity" value="True" />
       <add key="Sync.Crm.ReplacementRivaInstanceMachineId.RivaFS1" value="RivaDR1" />
     </appSettings>
   </configuration>

3. In the Riva CRM Service Monitor, restart the service, and let Riva complete at least one full sync cycle of all syncing users.

Recommended daily backup guidelines for Riva disaster recovery

Standard backup utilities can be used to create and schedule the daily backup of the primary Riva server folder structure. We recommend the following backup strategy:

• Automatically perform a scheduled daily full backup and store it offsite.
• Schedule the backups to occur during the Riva blackout period (by default, between 11 pm and 5 am).
• Set up an automated backup sequence that
  • performs a net stop on all Omni windows services,
  • saves a full backup of the Riva files and folders, and
  • performs a net start on all Omni windows services.
• Only the most recent three days of Riva backups need to be retained.

The backups must include these folders and optionally include these folders:

Backups should be checked daily to ensure that the backup files are valid and can be extracted.

If you are using Outlook Profile MAPI connections in Riva:

If you have created Riva Outlook Profile MAPI connections, consider exporting and storing the Outlook Profile that the Riva connection is using.

Document the Riva service settings:

Document the settings for Logon Identity and for Recovery Options.

To test the disaster recovery and data sync restoration procedures:

After the disaster recovery option has been initiated, Riva admins should document and test procedures for restoring data sync. The typical test scenario would involve:

1. On the primary Riva server, stop services, disable the Riva sync service, and shut down the host Windows system.

   This simulates a crash of that server.

2. Ensure that users continue to perform normal tasks in their mailbox and CRM user account.

3. Restore the passive fail-over Riva server:

   • Scenario 1 - Activate the Mirror Snapshot Fail-over Riva server: Enable the Riva sync service, and monitor data sync. Riva should remap previously synced data and then start syncing new data.

   • Scenario 2 - Activate the Passive Fail-over Riva server: Restore the latest Riva backup into the passive Riva fail-over server. Enable the Riva sync service and sync policy or policies, and monitor data sync. Riva should remap previously synced data and then start syncing new data.

   • Scenario 3 - Build and Activate a Replacement Fail-over Riva server: Build a new Windows host system that meets the requirements. Restore the latest Riva backup onto the new Windows host system. Riva sync service, and monitor data sync. Riva should remap previously synced data and then start syncing new data.

4. After data sync has resumed and been confirmed, stop services, and disable the Riva sync service.

5. Discontinue the use of the fail-over or replacement Riva server. Ensure that the Riva sync policies are disabled and the sync service is stopped and disabled.

6. Ensure that users continue to perform normal tasks in their mailbox and CRM user account.

7. Start the Windows system that hosts the primary Riva server. Start the Riva sync service, enable the Riva sync policy or policies, and monitor data sync. Riva should remap previously synced data and start syncing new data.

8. Document any challenges with the disaster recovery and data sync restoration procedures.

9. If required, revise the procedures and schedule additional procedure tests.

Applies to

• Riva On-Premise for Exchange (EWS). Note: MAPI is not supported.
• Riva On-Premise for Notes / Domino.