Migrate Module Inaccessible
Incident Report for Virtana
Postmortem

We experienced an issue where Migrate projects were not visible in the UI for all users due to an error that prevented the Migrate module from being accessed. The root cause for this was identified to be exhaustion in database resources due to scheduled cleanup activities that help in improvement in the performance of our platform and a large project upload occuring simultaneously.

To mitigate the issue, we scaled down the service which recovered the database and helped in restoring Migrate projects for all tenants. We also temporarily hid the Migrate module to ensure all Virtana users could successfully sign in while the database recovered. The module has since been unhidden and Migrate is accessible.

We are working on a permanent fix to ensure that the cleanup tasks are performed in a manner that does not result in the exhaustion of resources, particularly when large project uploads occur simultaneously.

We apologize for any inconveniences this incident may have caused.

Posted Jul 18, 2022 - 18:40 UTC

Resolved
This issue has been mitigated.
Optimize has been set as the default module and the migrate button has been hidden for now so all users are able to successfully sign in to Virtana.
We are continuing to work towards a long term solution. We apologize for any inconveniences in the meantime.
Posted Jul 16, 2022 - 02:08 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 15, 2022 - 23:41 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 15, 2022 - 22:29 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 15, 2022 - 21:43 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 15, 2022 - 21:05 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 15, 2022 - 20:31 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 15, 2022 - 19:19 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 15, 2022 - 18:39 UTC
Update
We are still waiting for results of the fix as it continues to implement.
Posted Jul 15, 2022 - 18:00 UTC
Update
An additional fix has been implemented. We are waiting to see the results.
Posted Jul 15, 2022 - 17:29 UTC
Update
Upon further review, the fix did have the expected result. We are waiting to see the impact it has towards resolution.
Posted Jul 15, 2022 - 17:02 UTC
Update
The fix that was implemented did not have the expected result.

We are investigating other options for a fix.
Posted Jul 15, 2022 - 16:52 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 15, 2022 - 16:44 UTC
Investigating
We are currently investigating an issue that prevents the Migrate module from being accessed.

Users may see an error message "Exceed limit of PG storage connections or database is not accessible" when they sign in to Virtana.

More updates to follow.
Posted Jul 15, 2022 - 16:36 UTC
This incident affected: App-US (US) (Data Access).