Availability issues with upstream provider impacting apps in US and EU regions

EU US
Apps 10 hours, 55 minutes

Activity

  • Resolved

    This issue is now resolved.

    Posted 3 years ago, Dec 8, 2021 03:27 UTC

  • Monitoring

    The engineers observe that metrics are within nominal ranges and keep monitoring them.

    Posted 3 years ago, Dec 8, 2021 02:17 UTC

  • Update

    Our third-party vendor is continuing to work on a number of different mitigation and resolution actions. They've identified the root cause of this issue to be an impairment of several network devices. While there are some early signs of recovery, there remains no ETA for full recovery yet.

    Posted 3 years ago, Dec 7, 2021 21:23 UTC

  • Update

    Multiple engineering teams from our third-party vendor are still actively working on parallel mitigation paths for this issue. While we are seeing incremental recovery, it is not consistent and there's still no ETA for full recovery.

    Posted 3 years ago, Dec 7, 2021 20:25 UTC

  • Update

    Our third-party vendor has partially mitigated the incident. Heroku continues to work closely with them for a full path to mitigation. There's no ETA for recovery at this time.

    Posted 3 years ago, Dec 7, 2021 19:08 UTC

  • Update

    Engineers have detected that our upstream provider is experiencing elevated error rates in the US region. This may impact dyno creation/formation, connect dashboard, database backups, maintenances for a subset of apps hosted in both the US and EU regions.

    Posted 3 years ago, Dec 7, 2021 17:28 UTC

  • Issue

    Engineers are looking into possible availability issues in the US and EU regions.

    Customers will see failed resource creation, backups and maintenances on a subset of databases.

    Posted 3 years ago, Dec 7, 2021 16:31 UTC

Current status