Resolved
On January 16th, 2011 at 10PM PST Heroku was notified of a security vulnerability by David E. Chen, a long-time customer. We deployed a fix to our production environment the following day, January 17th, 2011 at 2pm PST.
We have done extensive analysis and have no reason to suspect this vulnerability was exploited. However, we believe it is important to let the community know about the situation and what we are doing to prevent similar issues in the future. As a precaution, we are working with add-on providers to change all credentials. We also recommend that users should change any manually set credentials in their apps as well.
The vulnerability was a window through which an unauthorized user could potentially gain read-only access to an app's deployed code and configuration variables.
We confirmed the vulnerability, determining that it was introduced on December 28th. The underlying bug was fixed on Monday January 17th, the day after we learned about it. It is no longer possible to exploit this vulnerability. We do not believe that any customer data was accessed or changed. We have thoroughly audited our logs for that period and have found no evidence that anyone exploited this vulnerability.
Consistent with best practices for security incidents, to minimize the risk of a 0-day exploit, we waited 5 days to notify the community and work with our add-on providers on a precautionary mitigation plan.
We believe it is important to take all prudent steps to ensure the safety of apps. Heroku uses environment variables to provide configuration information to apps. These variables often include things like database passwords, API tokens, and credentials that are used to access add-ons or other third-party services. Although there is no evidence that these were compromised, we are taking additional steps to protect users.
Our add-on partners have been notified of the problem and advised to update the credentials for all Heroku apps. You can track the status of credential changes for all add-on providers here. We expect all add-on credentials to be updated within the next week. We have already started rolling credentials for all Heroku hosted PostgreSQL databases and expect to complete the update this evening.
The process of updating credentials will require restarting all apps. While we do not expect any apps will have issues with the update, if you do run into any issues please open an urgent support ticket at http://support.heroku.com.
Some apps may make use of hard-coded credentials in either their source code or manually set configuration variables. As a precautionary measure, we recommend that you update these credentials.
Some examples of hard-coded credentials may include:
We have enabled advanced releases on all apps for free for the next 2 weeks, providing rollback capabilities and a log of changes made to your app. To use, update to the latest gem (sudo gem update heroku
) and run heroku releases
.
If you need help or have further questions about the incident, contact us at http://support.heroku.com/
We are making several changes to our process and technology architecture in an effort to prevent this type of security regression in the future. First, we have introduced automated regression testing to specifically check for permission issues. Second, we have expanded our security audit review process for all changes on the platform. Third, we are increasing the frequency of both internal and external security reviews to help ensure that we are continually following the industry best practices. Finally, we are testing a new environment for isolating customer processes from one another that will provide a second layer of protection beyond filesystem permissions.
We want to thank David E. Chen for his contribution to our community by helping us to identify this issue and working with us to resolve it. Heroku is committed to continued improvements to our trust and transparency. Any individuals who believe they've identified a security issue within Heroku should contact us at security@heroku.com.