Resolved
This incident has been resolved.
The incident did not impact production deployments or domains on existing applications, though did impact those creating new apps, deleting old apps, and creating new custom domains for new/existing apps. All resolution and connectivity issues with our upstream DNS provider have been resolved. This impact end time would be 02:53 UTC on 5th October.
Monitoring
Our upstream DNS provider have applied a fix and our engineers are verifying connectivity.
During the impact period, we were seeing increased DNS resolution failures for custom domains and the default domain (*.herokuapp.com) in Common Runtime and Private Spaces across the US and EU regions. Existing domains are NOT affected.
Update
We are seeing increased DNS resolution failures for custom domains and the default domain (*.herokuapp.com) in Common Runtime and Private Spaces across the US and EU regions. Existing domains are NOT affected at this time.
Our engineers are continuously working with our upstream DNS provider who has identified a potential fix. We’ll provide an update in 60 minutes, or sooner if additional information becomes available. Customers may continue to see delays of up to 2 hours for creating new apps, deleting old apps, and creating new custom domains for new/existing apps.
Update
Engineers are still continuing to work with the upstream DNS provider to mitigate the issue. Customers may continue to see delays up to 2 hours for creating new apps, deleting old apps, and creating new custom domains for new/existing apps.
Update
Engineers are continuing to work with the upstream DNS provider who has identified a potential fix. We will continue to update in 30 minutes. Customers may continue to see delays up to 2 hours for creating new apps, deleting old apps, and creating new custom domains for new/existing apps.
Update
Engineers are continuing to work with upstream DNS provider to resolve the impact. Customers may continue to see delays up to 2 hours for creating new apps, deleting old apps, and creating new custom domains for new/existing apps.
Issue
Our engineering teams continue to work with upstream DNS provider for issues impacting propagation delay times. Customers could see delays up to 2 hours for creating new apps, deleting old apps, and creating new custom domains for new/existing apps.
Investigating
We are seeing increased failure rates to resolve DNS on recently created apps' default domains (*.heorkuapp.com) and recently created/updated custom domains in Common Runtime starting at 19:27 UTC. Existing domains are not affected at this time. Please be advised, changes to existing domains may result in DNS propagation delays at this time while the incident is being investigated.