On Monday, May 16th, beginning around 02:30 Pacific time our monitoring systems began detecting problems with some nodes in our HTTP reverse proxy tier. The primary on-call engineer was paged and quickly escalated the problem to the on-call incident commander who began i…
Resolved
We have successfully mitigated the ongoing DDoS attack. All app operations are fully restored to all applications.
Update
All tools operation including git and api usage is now fully restored.
Customers with properly configured DNS pointing to proxy.heroku.com should now have full app operations restored. Apps should ensure that their hostname is a CNAME record pointing to proxy.heroku.com. Root domains should continue to use A records. You can test if your hostmame is setup correctly by running:
$ host www.heroku.com www.heroku.com is an alias for proxy.heroku.com. $ host heroku.com heroku.com has address 50.19.85.132 ...
See the docs for full information on DNS setup.
Update
We are still experiencing intermittent connectivity issues for HTTP and Git traffic. These issues are still isolated to applications that have a root domain pointed to Heroku via static IP addresses.
We're continuing to work with our network provider to mitigate these availability issues.
Update
We're currently experiencing connectivity issues for both HTTP and Git traffic. The issues should be isolated only to applications that have a root domain pointed to Heroku via static IP addresses. Subdomains pointed to proxy.heroku.com, Hostname SSL setups, and IP SSL setups are all functioning normally.
We're working closely with our network service provider to mitigate availability issues stemming from what we believe to be a DDoS attack. The current attack protection procedures have reduced the effects of this attack to intermittent issues.
While we do not currently have an ETA for full resolution, connection error rates have fallen to very low levels. Our current plan is to leave the DDoS mitigation rules in place overnight. We will continue to monitor the situation and will resolve these intermittent issues as quickly as possible.
Update
We continue to work with our provider but have no new information to report at this time.
Update
We don't have any additional information to report at this time. The errors appear to be coming in at a low rate, and we're continuing to work with our provider to resolve them entirely. We will provide additional updates as information is available. Our next planned update will be at 10:00 Pacific Time.
Issue
We are currently working with our network provider to mitigate network problems that are affecting a small number of our front-end web servers. This is primarily affecting those customers that are pointing a root domain to Heroku via static IP addresses.