2020-11-25 Amazon Web Services Outage (Riva Cloud)

Service restored for Riva Cloud customers at 12:20 pm MDT on November 25, 2020.

Summary

AmazonAWS outage was affecting Riva Cloud login and logging which can also cause delayed sync cycles.  

Root Cause Analysis

After review, it was determined that the Amazon Web Services (AWS) Kinesis Data Streams was impaired in the US-EAST Region. 

Amazon had confirmed: This issue is also impacting other services, including ACM, Amplify Console, API Gateway, AppMesh, AppStream2, AppSync, Athena, AutoScaling, Batch, CloudFormation, CloudTrail, CloudWatch, Cognito, Connect, DynamoDB, EventBridge, IoT Services, Lambda, LEX, Managed Blockchain, Marketplace, Personalize, Resource Groups, SageMaker, Support Console, Well Architected, and Workspaces. This issue has also affected our ability to post updates to the Service Health Dashboard.

For more detail: https://status.aws.amazon.com/ 

Impact for Affected Customers

Riva Cloud was inaccessible for customers on our US datacenter and low-level failures affecting logging.  www.rivacloud.com

Riva Cloud Status updates: http://status.rivacloud.com/

Action Plan

Our Riva Cloud Operations team completed a failover to a stand-by Kinesis Data Stream in the US-EAST-2 AWS Region

Next Steps

We are continuously monitoring all systems to ensure the initial cause of the issue does not recur. We realize your successful daily customer interactions, sales, and support processes depend on a trusted system providing reliable, secure, and high-quality processing – this is our top priority. We sincerely apologize for any impact these disruptions may have caused.

If there are any outstanding questions, please do not hesitate to contact us.

Timeline

Technical Details

Note: All times are in Mountain Time (UTC-0600)

Service restored: Restored and service confirmed as healthy November 25, 2020 at 12:20 pm MDT
 

Wednesday, November 25, 2020

• 06:30 - Automated Riva Cloud environment alerted the team to the issue
• 07:37 - Issue identified as AWS, and ticket opened with AWS support
• 08:40 - Internal review of options with Riva Cloud Operations team
• 10:00 -  Working on failover to a stand-by Kinesis Data Stream in the US-EAST-2 AWS Region
• 12:20 - Failover to stand-by completed and Riva Cloud logging and auditing resumed