Yesterday 06/09/2021 at 8:15 AM UTC a failed deploy of one of our core Migrate services caused a database inconsistency which prevented project uploads between 8:15 AM UTC and 4:00 PM UTC. We rolled back the application change in order to resolve the issue and redeployed it successfully after the incident once we had verified a fix in our test environment. No data was lost during the incident.
We're taking action to improve our internal processes for incident management in an effort to identify and respond to them more swiftly in the future.
Posted Jun 10, 2021 - 17:01 UTC
Resolved
Issue has been resolved. New projects are getting uploaded fine. We will provide more details in the next 24 hours.
Posted Jun 09, 2021 - 16:09 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 09, 2021 - 16:04 UTC
Update
We are still working to resolve the issue.
Posted Jun 09, 2021 - 15:47 UTC
Update
We are stil working to resolve the issue.
Posted Jun 09, 2021 - 15:30 UTC
Identified
We've identified an issue where project uploads are failing and are working to resolve the issue.
Posted Jun 09, 2021 - 15:14 UTC
This incident affected: App-US (US) (Data Ingest).