A fix has been implemented and we are monitoring the results.
Posted Jun 13, 2023 - 18:31 EDT
Update
Update - AWS has communicated an ETA of around one hour for full restoration of services. Upon full AWS restoration of services, we will seamlessly revert our services back to standard mode
Posted Jun 13, 2023 - 18:12 EDT
Update
While AWS services are not fully restored, we are back to full operations using our workaround solutions. As soon as we receive a greenlight from AWS that all services are up and running, we will seamlessly revert back to standard configuration.
Posted Jun 13, 2023 - 17:38 EDT
Update
Update from AWS -
Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard.
Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support.
We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.
Posted Jun 13, 2023 - 16:57 EDT
Update
While the AWS outage is ongoing and their technicians are still investigating the issues, DW has failed over to form a workaround for selected services. Orders and most APIs should be functional at this time
Posted Jun 13, 2023 - 16:31 EDT
Update
Update From AWS:
We are continuing to experience increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. We have identified the root cause as an issue with AWS Lambda, and are actively working toward resolution.We are actively working on full mitigation and will continue to provide regular updates.