Between November 25, 2020 at 14:30 UTC and November 26, 2020 at 21:45 UTC, customers experienced errors or delays with several platform features including builds, CI, and actions related to dynos. We sincerely apologize for the negative effects our customers experienced.
P…
Resolved
Due to events that happened in this incident, Private Spaces may have experienced an inconsistent Dyno state, fewer than 10 Spaces may remain and engineers are running tasks to validate and correct those as necessary, they will all be resolved shortly.
This issue has been resolved.
Update
Our engineers continue to work on resolving the dyno state issue that is affecting very few customers.
We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 21:45:00 UTC (2020-11-26 16:45:00 EST)
Update
Our engineers continue fine-tuning the fix for the dyno state issue, and we expect conditions to keep improving as they proceed with their work.
We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 19:45:00 UTC (2020-11-26 14:45:00 EST)
Update
Our engineers continue to fix the dyno state issue for a few remaining customers. Customers may still see an inconsistent display of dyno state information and stuck jobs.
We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 17:45:00 UTC (2020-11-26 12:45:00 EST)
Update
Our engineers are working on fixing the dyno state issue for a few remaining customers. The rest of the platform continues to be healthy.
If you are seeing any issues related to dyno management, please reach out to our Support team.
We will continue to monitor and send the next update once everything has returned to normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 15:45:00 UTC (2020-11-26 10:45:00 EST)
Update
Our engineers continue working to fix the inaccurate dyno sync state issue (heroku ps
CLI command and dashboard potentially showing incorrect data).
We are keeping a check on the services and will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 13:45:00 UTC (2020-11-26 08:45:00 EST)
Update
We see a full recovery in most of the platform, including:
A few customers may see an inaccurate dyno sync state (heroku ps
CLI command and dashboard potentially showing incorrect data) and our engineers are working towards the resolution.
We will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 11:45:00 UTC (2020-11-26 06:45:00 EST)
Update
Our engineers have verified that the issue with launching one-off Dynos and restarting Dynos is resolving steadily as they continue remedial intervention and services return to normal conditions. We're seeing an improvement to affected services as we continue to work on a resolution.
We are keeping a check on the services and will post an update once everything is normal or in two hours(whichever will be earlier). We will provide next update on 2020-11-26 09:45:00 UTC (2020-11-26 4:45:00 EST)
Update
Our engineers are working on resolving this issue with our upstream provider and we can see favorable improvements.
Some customer should begin seeing things return to normal as things begin to recover.
We are keeping a check on the services and will post an update once everything is normal or in two hours.
We will provide next update on 2020-11-26 06:45:00 UTC (2020-11-26 1:45:00 EST)
Update
Our engineers continue to work on resolving this issue with our upstream provider and are starting to see improvements.
We are still seeing increased latency when provisioning new ACM certificates. Apps with recently-enabled ACM and apps with new custom domains using ACM may take longer to provision valid certificates.
To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.
We will provide next update on 2020-11-26 04:45:00 UTC (2020-11-25 23:45:00 EST)
Update
Our engineers continue to work on resolving this issue with our upstream provider. We expect this incident to continue for several more hours.
We are seeing increased latency when provisioning new ACM certificates. Apps with recently-enabled ACM and apps with new custom domains using ACM may take longer to provision valid certificates.
To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.
We will maintain our 2-hour update cadence. In the absence of new information, we will provide our next update at 2020-11-26 02:45:00 UTC (2020-11-25 21:45:00 EST).
Update
Our engineers continue to work on resolving this issue with our upstream provider. We expect this incident to continue for several more hours.
To avoid disruption to currently-running apps, customers should not run database maintenances and should reschedule any database maintenances scheduled to run today or tomorrow.
We will maintain our 2-hour update cadence. In the absence of new information, we will provide our next update at 2020-11-26 00:45:00 UTC (2020-11-25 19:45:00 EST).
Update
As our engineers continue to work on resolving this issue with our upstream provider, we expect this incident to continue for a number of hours.
We are moving to a 2 hour update cadence from our standard hourly updates. In the absence of new information, we will provide our next update at 10:45 UTC (5:45 Eastern Time).
Update
Heroku dynos are unable to be started, causing issues including the following platform features:
heroku run
, heroku pg:psql
, Heroku Scheduler, etc.)heroku ps:scale
/ heroku ps:resize
) will not be able to create new dynosgit push heroku main
, Heroku GitHub deploys, container deploys, release phase, etc.)We highly recommend avoiding any of the above, as well as changing environment variables, provisioning add-ons, etc. as these may cause your app to become unavailable.
At this time, we are unable to assist with any one-off assistance when apps are unavailable due to this incident.
Update
Our engineers are still continuing to investigate this issue.
Update
This issue is also affecting Heroku schedulers, pipelines, review apps and Heroku connect.
Engineers are continuing to investigate this.
Update
Heroku customers are continuing to experience issues with One-off dynos, Dyno scaling, Dyno restarts, Builds, Heroku CI runs.
While we continue to investigate this issue, it is recommended not to perform dyno operations like restarts, deploys, and anything that affects the dyno state.
Issue
We are seeing issues with new dynos starting on the platform.
This is causing issues including the following platform features:
One-off dynos Dyno scaling Dyno restarts Builds Heroku CI runs
Our engineers are investigating the issue.
Investigating
Engineers are investigating potential issues with the Heroku platform.