Migrate project are not being displayed in the UI

Incident Report for Virtana

Postmortem

The Virtana Platform experienced an issue where migration projects for each tenant could not be viewed in the UI. The root cause was database exhaustion caused by a scheduled migration task, an effort towards a continued performance improvement in our platform.

To mitigate the issue, we turned off the feature flag controlling this new capability. This prevented the service from scheduling migration jobs, thus allowing the database to recover and release resources required to serve up migration projects for each tenant.

We are also introducing a permanent fix to ensure that such migration jobs are only executed after hours.

Posted Apr 07, 2022 - 15:12 UTC

Resolved

This incident has been resolved.

Posted Apr 07, 2022 - 00:09 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Apr 06, 2022 - 23:40 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Apr 06, 2022 - 23:00 UTC

Investigating

We have identified an issue where all migrate projects for all tenants are not being displayed in the UI. We are currently investigating the issue and apologize for any inconvenience.

Posted Apr 06, 2022 - 22:25 UTC

This incident affected: App-US (US) (Data Ingest) and Data Ingest.