Starting last Thursday, Heroku suffered the worst outage in the nearly four years we've been operating. Large production apps using our dedicated database service may have experienced up to 16 hours of operational downtime. Some smaller apps using shared databases may have experienced up to 60 hou…
Resolved
We have fully restored git services for all applications and App Operations are functioning as expected. We will provide a postmortem for this incident once we've completed our internal review.
Update
At this time all shared databases and all dedicated databases are restored. App Operations are functioning as expected.
We're still working to restore git push functionality to a very small percentage of applications and expect to have this functionality restored soon.
Update
At this time most applications are online and we're making progress recovering the remaining affected shared databases. We do not have an ETA but the pace of recovery has increased.
We're still working to restore deploy functionality for all applications.
Update
At this time most applications are online. Deploys have been restored for a majority of applications too. We're still hard at work recovering the remaining affected shared databases and deploy functionality. We do not at this time have an ETA but are dedicated to fully restoring all functionality as quickly as possible.
Update
We are continuing to work on recovering shared databases and restore git push functionality to any affected applications with all available resources. We do not at this time have an ETA.
Update
We have successfully recovered a number of databases. Additional applications running on a few shared database servers are still being recovered. We are continuing to work on recovering these shared databases with all available resources. We do not at this time have an ETA for the full recovery of these shared databases.
Update
We are aware of the incredible difficulties this downtime is causing many customers.
Current Operational Status:
All dedicated database applications are fully online.
The majority of shared databases are online.
The majority of applications can deploy via git.
New app creation is fully working.
Next steps:
We are working with our service provider to restore both deployments and operation to affected shared databases as quickly as possible. In parallel, we are working on alternative recovery options.
Update
We have restored all dedicated databases and are continuing to work on the effected shared databases and deploy tools.
Update
We are still working through restoring the affected databases and restoring full deploy capabilities.
Update
We have successfully brought up our core services and begun restoring service to applications. Many applications are now fully operational. The remaining effected apps databases are being brought online now. We will continue to work to bring the remainder of applications up as quickly as possible.
API services are now fully restored, and all gem commands are now working. Deploys are working for some applications. We will continue to work on restoring deploys for the remaining applications.
Update
We have successfully brought up our core services and begun restoring service to applications. Many applications are now fully operational. The remaining effected apps databases are being brought online now. We will continue to work to bring the remainder of applications up as quickly as possible.
API services are now fully restored, and all gem commands are now working. We are working on restoring deploys.
Update
We are continuing to restore service to applications. In some cases the process of bringing many applications online simultaneously has created intermittent availability and elevated error rates. We continue to work to fully restore availability as quickly as possible.
Update
We have successfully brought up our core services and begun restoring service to applications. Many applications are now fully operational. We will continue to work to bring the remainder of applications up as quickly as possible and then restore api and git services.
Update
We have been able to successfully boot new servers and are in the process of restoring our core services. Once our core services come online we will be able to start to bring app operations back online. We will post further updates as soon as we have additional information.
Update
We continue to experience widespread connectivity issues that are preventing us from booting servers. We are working with our service provider to resolve this as soon as possible. We do not currently have an estimate for when this will be resolved. We will post further updates as soon as we have additional information.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
We are continuing to work with our service provider to restore outstanding connectivity issues. We will continue to update every half hour or as new information becomes available.
Update
There is nothing new to report. We're continuing to work to get full connectivity restored and will continue updating every half hour.
Update
There is nothing new to report. We're continuing to work to get full connectivity restored and will continue updating every half hour.
Update
There is nothing new to report. We're continuing to work to get full connectivity restored and will continue updating every half hour.
Update
There is nothing new to report. We're continuing to work to get full connectivity restored and will continue updating every half hour.
Update
We're still seeing elevated error rates due to connectivity issues and are working with our service provider to fully restore service.
Update
We're still seeing elevated error rates due to connectivity issues and are working with our service provider to fully restore service.
Update
We're still seeing elevated error rates due to connectivity issues and are working with our service provider to fully restore service.
Update
We're still seeing elevated error rates due to connectivity issues and are working with our service provider to fully restore service.
Update
We're still seeing elevated error rates due to connectivity issues and are working with our service provider to fully restore service.
Update
Connectivity issues are causing applications and tools to work intermittently. We're working with our network service provider to fully restore service. There is nothing new to report at this time.
Update
Connectivity issues are causing applications and tools to work intermittently. We're working with our network service provider to fully restore service at this time.
Update
We do not have anything new to report at this time. We're still working with our network service provider to fully restore connectivity. Applications and tools are working intermittently at this time. We'll continue to update on a 30 minute interval unless we have something new to report.
Update
We're continuing to work with our network service provider to fully restore connectivity. Applications and tools are working intermittently at this time.
Update
The elevated error rates are due to connectivity issues. We're continuing to work with our network service provider to fully restore connectivity. Applications and tools are working intermittently at this time.
Update
Error rates appear to have stabilized. Both applications and tools are functioning as expected at this time. We're continuing to keep a close eye on the situation as we investigate the root cause.
Issue
We are investigating high error rates. We'll post an update when we know more.