[Service restored] Partial DNS LBR resolution failure in certain regions, work around in notes.
Incident Report for F5 Cloud Services
Resolved
Monitoring has shown no significant latency impact on our services and as such this incident is being marked as resolved. We will be providing a service update in the coming days to further address the problem identified by this incident. At this time the system is operational with no significant impact beyond optimal performance.
Posted Jul 21, 2021 - 16:37 PDT
Update
This issue has been fully identified and a complete resolution is being worked on. While we continue to work towards a resolution we wanted to provide an initial answer to what happened.

The US East regional DNS server experienced a network split forcing a cluster restart. The data persistence at this site was lost and was unable to recover without manual intervention. Due to the network split and following cluster restart the configuration replay mechanism was failing to automatically push configurations to this site. The workaround provided triggered the configuration to be pushed again to all regions fixing the problem. LBR queries answered at this site were temporarily impacted. Temporarily, the US East region has been taken out of the global DNS network while a fix is created to restore this data for all configurations.

We are working to return US East to production as soon as possible to return services to a fully operational state and reduce any additional latency introduced by its removal. Going forward, a more robust replay mechanism will be established from our control plane to restore lost data persistence without impact to customers.
Posted Jul 21, 2021 - 10:22 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 21, 2021 - 08:22 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 20, 2021 - 22:55 PDT
Monitoring
We have found and implemented a work around. All LBR queries are responding correctly. The root cause is known and a we are implementing permanent fix. During this time some LBR queries may experience a higher than expected latency. Once the permanent fix is available normal latencies will return.

Thank you for your patience while we implement the permanent fix.
Posted Jul 20, 2021 - 21:33 PDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 20, 2021 - 19:59 PDT
Identified
The F5 Cloud Services Team has identified the root cause and is working on resolving the issue.

Thank you in advance for your patience while we are taking urgent actions to restore the service.
Posted Jul 20, 2021 - 19:27 PDT
Investigating
We are currently investigating an issue with DNS query resolution for Load Balanced records. The issue appears isolated to specific regions. More info as we investigate.

Work around: If you are experiencing resolution failure the following steps should resolve the issue while we make a permanent fix.

For the affected LBRs:
1. Navigate to the impacted zone
2. Make an inconsequential update to the zone, such as adding something in the LBR notes.
3. Save the zone. You may need to click ‘save’ if in a specific component, then ‘save’ for the zone.
Posted Jul 20, 2021 - 19:18 PDT
This incident affected: DNS.