May 10, 00:27 UTC
Update - We have verified that our remedial actions have been effective in restoring the operations of apps that were affected by this incident. During our cross-examination, we haven't found any evidence of errors impacting the consistency of the data stored in our database. Later today we will re-run the migration of custom entities data to ensure no entry contains an empty value. This process is estimated to finish in the next 2-3 days. Once this is done we will close the incident.
We expect to have an update within the next 3-4 days.
May 7, 06:11 UTC
Update - We have remediated the custom entity data store for the remaining identified sites. The read access to the custom entities data store has been restored to a state that no longer returns an empty string. We are monitoring and cross-checking the errors recorded on our platform to ensure the integrity of the data and the migration process.
We expect to have an update within the next 24-48 hours.
May 6, 07:51 UTC
Monitoring - We have restored read access to custom entity data stores for 99.5% of identified sites.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
May 6, 07:40 UTC
Identified - We have identified an issue with Forge apps that integrate the custom entity data store. While reading historical data that has not been updated since April 5th, the service may return an empty string instead of the value stored in the database. This would have appeared to end users as though the application data was missing. This was due to a logic error resulting in incomplete migration of data during our current data residency rollout.
Timeline
2024-04-30 06:15 UTC: Rollout of feature flag to read custom entity data from regional data. stores as per data residency setting for the workspace.
2024-05-05 23:00 UTC: Incident raised after partners reported errors.
2024-05-06 05:00 UTC: Root cause detected.
2024-05-06 06:00 UTC: Remediation steps approved.
We expect to resolve approximately 99.5% of impacted data to be available within the next 10 minutes.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
May 6, 04:09 UTC
Update - We are continuing to investigate this issue.
May 6, 04:09 UTC
Investigating - We have identified a problem with querying custom entities where a query returns an empty string for some keys instead of the value stored in the database. At this time we believe that the issue is limited to query operations in a small number of apps and all the other operations are not impacted. We are investigating the root cause and will provide another update by 08:00 UTC.