Issues with starting new dynos

US
Apps 30 hours, 7 minutes

Follow-up Report

Activity

  • Resolved

    Due to events that happened in this incident, Private Spaces may have experienced an inconsistent Dyno state, fewer than 10 Spaces may remain and engineers are running tasks to validate and correct those as necessary, they will all be resolved shortly.

    This issue has been resolved.

    Posted 4 years ago, Nov 26, 2020 21:45 UTC

  • Update

    Our engineers continue to work on resolving the dyno state issue that is affecting very few customers.

    We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 21:45:00 UTC (2020-11-26 16:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 19:41 UTC

  • Update

    Our engineers continue fine-tuning the fix for the dyno state issue, and we expect conditions to keep improving as they proceed with their work.

    We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 19:45:00 UTC (2020-11-26 14:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 17:44 UTC

  • Update

    Our engineers continue to fix the dyno state issue for a few remaining customers. Customers may still see an inconsistent display of dyno state information and stuck jobs.

    We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 17:45:00 UTC (2020-11-26 12:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 15:49 UTC

  • Update

    Our engineers are working on fixing the dyno state issue for a few remaining customers. The rest of the platform continues to be healthy.

    If you are seeing any issues related to dyno management, please reach out to our Support team.

    We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 15:45:00 UTC (2020-11-26 10:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 13:47 UTC

  • Update

    Our engineers continue working to fix the inaccurate dyno sync state issue (heroku ps CLI command and dashboard potentially showing incorrect data).

    We are keeping a check on the services and will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 13:45:00 UTC (2020-11-26 08:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 11:45 UTC

  • Update

    We see a full recovery in most of the platform, including:

    • One-off dynos, dyno scaling, and scheduler
    • Releases, pipelines, and CI
    • Postgres backups and maintenances

    A few customers may see an inaccurate dyno sync state (heroku ps CLI command and dashboard potentially showing incorrect data) and our engineers are working towards the resolution.

    We will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 11:45:00 UTC (2020-11-26 06:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 09:45 UTC

  • Update

    Our engineers have verified that the issue with launching one-off Dynos and restarting Dynos is resolving steadily as they continue remedial intervention and services return to normal conditions. We're seeing an improvement to affected services as we continue to work on a resolution.

    We are keeping a check on the services and will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 09:45:00 UTC (2020-11-26 4:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 07:58 UTC

  • Update

    Our engineers are working on resolving this issue with our upstream provider and we can see favorable improvements.

    Some customer should begin seeing things return to normal as things begin to recover.

    We are keeping a check on the services and will post an update once everything is normal or in two hours.

    We will provide next update on 2020-11-26 06:45:00 UTC (2020-11-26 1:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 05:21 UTC

  • Update

    Our engineers continue to work on resolving this issue with our upstream provider and are starting to see improvements.

    We are still seeing increased latency when provisioning new ACM certificates. Apps with recently-enabled ACM and apps with new custom domains using ACM may take longer to provision valid certificates.

    To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.

    We will provide next update on 2020-11-26 04:45:00 UTC (2020-11-25 23:45:00 EST)

    Posted 4 years ago, Nov 26, 2020 03:33 UTC

  • Update

    Our engineers continue to work on resolving this issue with our upstream provider. We expect this incident to continue for several more hours.

    We are seeing increased latency when provisioning new ACM certificates. Apps with recently-enabled ACM and apps with new custom domains using ACM may take longer to provision valid certificates.

    To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.

    We will maintain our 2-hour update cadence. In the absence of new information, we will provide our next update at 2020-11-26 02:45:00 UTC (2020-11-25 21:45:00 EST).

    Posted 4 years ago, Nov 26, 2020 00:45 UTC

  • Update

    Our engineers continue to work on resolving this issue with our upstream provider. We expect this incident to continue for several more hours.

    To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.

    We will maintain our 2-hour update cadence. In the absence of new information, we will provide our next update at 2020-11-26 00:45:00 UTC (2020-11-25 19:45:00 EST).

    Posted 4 years ago, Nov 25, 2020 22:39 UTC

  • Update

    As our engineers continue to work on resolving this issue with our upstream provider, we expect this incident to continue for a number of hours.

    We are moving to a 2 hour update cadence from our standard hourly updates. In the absence of new information, we will provide our next update at 10:45 UTC (5:45 Eastern Time).

    Posted 4 years ago, Nov 25, 2020 20:40 UTC

  • Update

    Heroku dynos are unable to be started, causing issues including the following platform features:

    • One-off dynos are unable to be created (including heroku run, heroku pg:psql, Heroku Scheduler, etc.)
    • Dyno scaling (heroku ps:scale / heroku ps:resize) will not be able to create new dynos
    • Dyno restarts will cause the existing dyno to be stopped but will not replace the dyno
    • Builds are unable to complete (including git push heroku main, Heroku GitHub deploys, container deploys, release phase, etc.)
    • Heroku CI runs will not be created
    • Heroku PGBackups tasks will not be created

    We highly recommend avoiding any of the above, as well as changing environment variables, provisioning add-ons, etc. as these may cause your app to become unavailable.

    At this time, we are unable to assist with any one-off assistance when apps are unavailable due to this incident.

    Posted 4 years ago, Nov 25, 2020 20:05 UTC

  • Update

    Our engineers are still continuing to investigate this issue.

    Posted 4 years ago, Nov 25, 2020 19:07 UTC

  • Update

    This issue is also affecting Heroku schedulers, pipelines, review apps and Heroku connect.

    Engineers are continuing to investigate this.

    Posted 4 years ago, Nov 25, 2020 18:11 UTC

  • Update

    Heroku customers are continuing to experience issues with One-off dynos, Dyno scaling, Dyno restarts, Builds, Heroku CI runs.

    While we continue to investigate this issue, it is recommended not to perform dyno operations like restarts, deploys, and anything that affects the dyno state.

    Posted 4 years ago, Nov 25, 2020 17:17 UTC

  • Issue

    We are seeing issues with new dynos starting on the platform.

    This is causing issues including the following platform features:

    One-off dynos Dyno scaling Dyno restarts Builds Heroku CI runs

    Our engineers are investigating the issue.

    Posted 4 years ago, Nov 25, 2020 16:12 UTC

  • Investigating

    Engineers are investigating potential issues with the Heroku platform.

    Posted 4 years ago, Nov 25, 2020 15:37 UTC

Current status