Heroku Status

Current Status and Incident Report

Ruby deploys temporarily degraded

Development 14h

Resolved

After an extensive review of the RubyGems archive, conducted in conjunction with multiple independent mirrors, we are re-enabling fetching of remote gems during pushes. RubyGems itself remains in a partially degraded functionality, but all Heroku developers should now be able to update their Gemfiles. Any developers who used the temporary BUILDPACK_URL workaround are encouraged to revert it with heroku config:unset BUILDPACK_URL --app appname.

For more information on RubyGems status, see http://status.rubygems.org/

Update

The RubyGems team has verified that about 84% of all gems stored at RubyGems are unmodified. An external 2-month old mirror was used for the verification, and the gap is due to gems pushed in the last two months. A second mirror is currently being used to cover the remaining 16%.

Once this is complete, we hope to restore full push functionality.

Update

While the RubyGems team is continuing to investigate audit logs and compare all gems against external known-good copies, there has been no evidence yet that any gems have been malicious modified. As a precaution, Ruby deploys that require external gem servers continue to be disabled.

Update

We have enabled Ruby deploys for applications that can install all of their gems from prior deployments. If an application has not changed their Gemfile or Gemfile.lock, gems will be installed from the local cache and the deploy will succeed, so application changes can safely be deployed. However, if accessing rubygems.org (or another remote gem source) is required to resolve dependencies or install new gems, the deploy will be rejected. We are coordinating with the rubygems.org team, and will re-enable remote gem fetches once all parties are satisified that the incident has been resolved and no gems have been maliciously modified.

Update

Rubygems.org was hacked due to an YAML parsing vulnerability. At least one malicious gem was uploaded which potentially had access to sensitive data, including credentials necessary to tamper with gems.

Currently the rubygems.org team is verifiying all gems since it’s unknown which have been tampered with. This will be an incremental process whereby they will start with the latest versions of all gems, then all versions of the most popular 100 gems, then the next 1000, and finally all of them.

We have disabled deploys of ruby applications until we gain confidence that no gems have been compromised. Users wishing to work around this can deploy at their own risk by setting a custom BUILDPACK_URL as shown in the instructions on GitHub. However, we strongly discourage its use until we have determined the authenticity of all gems.

Issue

Security Notice

Rubygems.org has been affected by a recent YAML parsing vulnerability. Ruby deploys have been temporarily disabled to protect our users from malicious gems. We will have more information available shortly, including a workaround for those who wish to deploy anyway.

Thus far, there is nothing to suggest that any widely used gems have been altered.

We're working to audit Rubygems changes and will have updates throughout the day.

← Current Status