Heroku Security Notification

Apps 169 hours, 43 minutes
Tools 935 hours, 36 minutes

Follow-up Report

Activity

  • Resolved

    Since being notified of this issue by GitHub on April 13, 2022, our security and engineering teams have worked diligently to investigate and mitigate the threat.

    Our investigation has concluded, and as previously mentioned, we restored the GitHub integration on May 25, 2022. We have no evidence of customer impact beyond what has already been reported, including no additional evidence of threat actor activity after April 14, 2022. We will publish details of the attacker’s actions on status.heroku.com the week of June 13, 2022.

    Finally, thank you for your patience, understanding, and feedback during this process. We acknowledge and regret any inconvenience you may have experienced. We appreciate your trust in us as we continue to make your success our top priority.

    Posted 3 years ago, May 31, 2022 23:52 UTC

  • Update

    We are happy to report that the GitHub integration is re-enabled! You can now reconnect with GitHub and restore your Heroku pipeline functionality, including Review Apps, with newly generated tokens.

    You can connect to GitHub immediately or wait for the enhanced integration as described in this blog post. To re-establish your GitHub connection now, please follow these instructions.

    Please continue to visit status.heroku.com for updates as they become available.

    Posted 3 years ago, May 25, 2022 19:52 UTC

  • Update

    We are dedicated to ensuring the security of our customers, and as such, continue to pursue all potential leads as part of our investigation. As part of that commitment, we want to let you know about two additional findings that may impact a small number of our customers.

    As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. Additionally, another small subset of Heroku users had their Heroku tokens exposed in a config var for a pipeline. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022.

    Any users affected by these issues were notified directly and provided with additional guidance. If you did not receive an email directly from us, we have no evidence that you were impacted by either of these recently identified issues. If you received an email from Salesforce (techcomms@mail.salesforce.com) and have identified suspicious activity, please contact security@salesforce.com.

    For an update on our current progress with regards to the GitHub integration, please see this blog post.

    Posted 3 years ago, May 19, 2022 13:17 UTC

  • Update

    Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved.

    • At this time we have seen no additional OAuth tokens compromised beyond what was reported on April 15, 2022.
    • GitHub has contacted all customers they identified as affected by the issue.
    • Heroku completed the necessary password resets on May 5, 2022.
    • We have no evidence of any unauthorized access to Heroku systems since April 14, 2022.
    • In the event we notify customers directly, the email communication will be sent from Salesforce (techcomms@mail.salesforce.com).

    We know you are waiting for us to re-enable our integration with GitHub, and we've committed to you that we will only do so following a security review. We will post more information to status.heroku.com when it is available.

    Posted 3 years ago, May 18, 2022 00:38 UTC

  • Update

    As our investigation continues into the weekend, we wanted to share a blog post from Heroku General Manager and Salesforce EVP, Bob Wise, regarding this issue and our response.

    Since our last update, we confirmed that Heroku has completed the necessary password resets. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022. This analysis is based on our investigation to date, backed by a leading third-party security vendor and our extensive threat detection systems.

    We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.

    We will continue to post updates to status.heroku.com as additional information becomes available. We appreciate your continuous feedback and encourage you to keep sharing questions with Heroku Support.

    Posted 3 years ago, May 7, 2022 00:29 UTC

  • Update

    We value transparency and understand our customers are seeking a deeper understanding of the impact of this incident and our response to date.

    We continue to work diligently in response to this Heroku incident first announced on April 15, 2022. We worked with GitHub, our threat intelligence vendors, other industry partners, and have been in touch with law enforcement to assist in our investigation. Without compromising our ongoing investigation or the security of our customers, we are able to share the following details.

    On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.

    GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. We remain committed to ensuring the integration is secure before we re-enable this functionality.

    Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.

    Please continue to visit status.heroku.com for updates as they become available.

    Posted 3 years ago, May 5, 2022 02:21 UTC

  • Update

    A subset of Heroku customers will receive email notifications directly from Salesforce Incident Alerts (incidentalerts@msg.salesforce.com) regarding our continuous efforts to enhance security. If you have any questions or need to address specific integration issues, please contact Heroku Support. We appreciate your collaboration and trust as we continue to make your success our top priority.

    Posted 3 years ago, May 3, 2022 23:58 UTC

  • Update

    As our investigation continues, we will be contacting customers directly to take any necessary security measures. Please continue to visit status.heroku.com for new information and guidance as it becomes available.

    Posted 3 years ago, May 2, 2022 23:07 UTC

  • Update

    We continue to make progress in our investigation. Please see previous status.heroku.com posts for information on how to request logs from GitHub, how to deploy to Heroku, and other common questions.

    Posted 3 years ago, May 1, 2022 16:44 UTC

  • Update

    Support and Security teams continue to respond to customer inquiries through Heroku Support. As a part of our effort to track indicators of compromise, we also encourage customers who have obtained logs from GitHub to share them with us by contacting security@salesforce.com. We appreciate our customers’ trust and collaboration as we continue to investigate this issue.

    Posted 3 years ago, Apr 30, 2022 15:34 UTC

  • Update

    Our investigation into this issue will continue through the weekend. Please monitor this site for updates and direct any inquiries you have to Heroku Support. We appreciate our customers’ trust in us as we work diligently to address this issue.

    Posted 3 years ago, Apr 29, 2022 22:24 UTC

  • Update

    Thank you for your patience as we continue to work through this issue. Our investigation is ongoing, and Support and Security teams continue to respond to customer inquiries. We appreciate all of the information received to date, and encourage you to keep sharing questions with Heroku Support and IOCs with security@salesforce.com.

    Posted 3 years ago, Apr 29, 2022 00:31 UTC

  • Update

    As our investigation continues, we understand our customers may receive notifications from GitHub directly. If you have obtained logs from GitHub that reveal potential indicators of compromise (IOCs), we encourage you to share those IOCs with us in an email to security@salesforce.com. If you need to address specific integration issues, please contact Heroku Support. We appreciate your collaboration and trust as we continue to make your success our top priority.

    Posted 3 years ago, Apr 27, 2022 17:07 UTC

  • Update

    For the protection of our customers, we will not be reconnecting to GitHub until we are certain that we can do so safely, which may take some time. We recommend that customers use alternate methods rather than waiting for us to restore this integration.

    The simplest approach to deploy Heroku apps is directly using git push heroku main. Alternatively, you can refer to one of the add-ons listed in the Continuous Integration and Delivery section of the Heroku Ecosystem Marketplace or other third party CI/CD products that offer Heroku support, such as Circle CI. You can also use community-maintained integrations such as this GitHub Action for deploying to Heroku.

    We sincerely regret any inconvenience you may have experienced because of this issue and appreciate your trust in us as we continue to make your success our top priority. Please continue to visit status.heroku.com for the latest updates.

    Posted 3 years ago, Apr 26, 2022 23:54 UTC

  • Update

    As a reminder, Salesforce Security does not have access to customer GitHub repository logs that would reveal what actions, if any, were taken by the threat actor, as the logs belong to GitHub and you as a GitHub customer.

    We recommend conducting a comprehensive review of your GitHub logs for evidence of exfiltration or malicious activity. If you discover logs that reveal indicators of compromise (IOCs), we encourage you to share them with us in an email to security@salesforce.com.

    To request logs from GitHub:

    Customers who are directly contacted by GitHub regarding this issue are welcome to contact GitHub according to directions in the notification received. For other questions regarding GitHub and npm, please contact GitHub Support.

    If you see evidence of exfiltration in your logs, you should look carefully through your repositories for any credentials that may have been compromised and mitigate access by disabling accounts and rotating credentials as needed. We also recommend revoking or rotating any exposed credentials. Source code credential scanning is one of the primary ways that malicious actors can escalate their access. Additionally, if you identify unusual activity in your logs, please review for the manipulation of your stored code.

    Posted 3 years ago, Apr 25, 2022 18:19 UTC

  • Update

    Our investigation is ongoing, and our Support and Security teams are continuing to respond to customer inquiries. As we work through this issue together, please contact Heroku Support to address specific integration issues or security@salesforce.com if you have obtained logs from GitHub and identified suspicious activity. We appreciate your patience as we make your success our top priority.

    Posted 3 years ago, Apr 24, 2022 18:02 UTC

  • Update

    As we head into the weekend, we remain vigilant and committed to our investigation. We will keep our customers apprised of new information and guidance as it becomes available. Please see previous status.heroku.com posts for information on how to request logs from GitHub and how to deploy to Heroku.

    Posted 3 years ago, Apr 23, 2022 18:08 UTC

  • Update

    We take the protection of our customers very seriously, and as a result, we will not be reconnecting to GitHub until we are certain that we can do so safely, which may take some time. We recommend that customers use alternate methods rather than waiting for us to restore this integration.

    The simplest approach to deploy Heroku apps is directly using git push heroku main. Alternatively, you can refer to one of the add-ons listed in the Continuous Integration and Delivery section of the Heroku Ecosystem Marketplace or other third party CI/CD products that offer Heroku support, such as Circle CI. You can also use community-maintained integrations such as this GitHub Action for deploying to Heroku.

    We sincerely regret any inconvenience you may have experienced as a result of this issue and appreciate your trust in us as we continue to make your success our top priority. Please continue to visit status.heroku.com for the latest updates.

    Posted 3 years ago, Apr 23, 2022 00:16 UTC

  • Update

    As our investigation continues, we understand our customers may receive notifications from both Heroku and GitHub directly, and we’re here to support. We take the protection of our customers’ data very seriously and are committed to providing necessary guidance during this critical time.

    While our customers remain unable to reconnect to GitHub via the Heroku dashboard, we wanted to share a supplement to the code deployment methods previously provided. For instructions on how to change your deployment method from GitHub to Heroku Git, please refer to the following Help article: How to switch deployment method from GitHub to Heroku Git with all the changes/app code available in a GitHub repo.

    Once we have determined that users can safely reconnect to GitHub via the Heroku dashboard, we will restore the integration. Please continue to visit status.heroku.com for updates.

    GitHub information that is only available to Heroku customers can be valuable in helping us understand the scope of this issue's impact. If you have obtained logs from GitHub and identified suspicious activity that you believe may assist our investigation, please contact security@salesforce.com.

    We sincerely regret any inconvenience you have experienced as a result of this incident and appreciate your collaboration and trust as we continue to make your success our top priority.

    Posted 3 years ago, Apr 21, 2022 23:53 UTC

  • Update

    We continue to make progress on our investigation into this issue. During the course of our investigation, information we have received from Heroku customers has been useful. If you have obtained logs from GitHub and identified suspicious activity that you believe may assist us in our investigation, please contact security@salesforce.com. We appreciate your collaboration and trust as we continue to make your success our top priority.

    We will continue to post updates to status.heroku.com as additional information becomes available.

    Posted 3 years ago, Apr 20, 2022 01:09 UTC

  • Update

    We are continuing to investigate this incident and have addressed our customers’ most common questions below.

    How can I get additional guidance for investigating GitHub logs for evidence of exfiltration or malicious activity? We understand that GitHub began notifying potentially affected customers with the following guidance. Customers who are directly contacted by GitHub regarding this issue are welcome to contact them according to directions in the notification received. For other questions regarding GitHub and npm, please contact GitHub Support.

    • Review all your private repositories for secrets or credentials stored in them. There are several tools that can help with this task such as GitHub secret scanning and trufflehog.
    • Review the OAuth applications that you’ve authorized for your personal account or that are authorized to access your organization and remove anything that’s no longer needed.
    • Follow GitHub’s guidelines for hardening the security posture of your GitHub organization.
    • Review your account activity, personal access tokens, OAuth apps, and SSH keys for any activity or changes that may have come from the attacker.

    What code deployment methods are available while we’re unable to reconnect to GitHub via the Heroku dashboard?

    Which Heroku features have become non-operative due to the removal of the Heroku-GitHub integration?

    • Enabling review apps
    • Creating (automatic and manual) review apps
    • Deploying (automatic and manual) review apps
    • Deploying an app from GitHub (either manual or automatic)
    • Heroku CI cannot create new runs (automatically or manually) or see GitHub branch list
    • Heroku Button: unable to create button apps from private repositories
    • ChatOps: unable to deploy or get deploy notifications
    • Any app with a GitHub integration may be affected by this issue. To address specific integration issues, please open a case with Heroku Support

    How and when will we be notified of the restoration of the Heroku-GitHub integration? We will provide updates to status.heroku.com as additional information becomes available.

    Posted 3 years ago, Apr 19, 2022 00:39 UTC

  • Update

    As reported yesterday, revocation of all OAuth tokens from the Heroku Dashboard GitHub integration is complete. Until further notice, we will not issue OAuth tokens from the Heroku Dashboard. These actions, based on our current understanding of the issue, should prevent unauthorized access to your GitHub repositories.

    We will continue to work with GitHub to provide additional guidance on how to review your GitHub logs for evidence of exfiltration or malicious activity. Please reach out to security@salesforce.com with any information that may assist us with our ongoing investigation.

    Please continue to visit status.heroku.com for the latest updates.

    Posted 3 years ago, Apr 17, 2022 18:58 UTC

  • Update

    Subject: Heroku Security Update: OAuth token revoked

    At 5:00 p.m. PT on April 16, 2022, Salesforce completed the revocation of all OAuth tokens from the Heroku Dashboard GitHub integration. As mentioned previously, this will prevent you from deploying your apps from GitHub through the Heroku dashboard or via Heroku automation, and some other actions in the dashboard will no longer work. While you will be unable to reconnect to GitHub via the Heroku dashboard, you may continue to use other code deployment methods available in the following documentation:

    Please continue to visit status.heroku.com for additional information as it becomes available.

    Posted 3 years ago, Apr 17, 2022 01:59 UTC

  • Update

    Heroku Security Update: OAuth token revocation

    Salesforce has made significant progress with the revocation of all OAuth tokens from the Heroku Dashboard GitHub integration. Revocation of the remaining OAuth tokens is our top priority and we will provide an update on this as soon as it is completed.

    As mentioned previously, this will prevent you from deploying your apps from GitHub through the Heroku dashboard or via Heroku automation, and some other actions in the dashboard will no longer work. While you will be unable to reconnect to GitHub via the Heroku dashboard, you may continue to use other code deployment methods available in the following documentation:

    Please continue to visit status.heroku.com for additional information as it becomes available.

    Posted 3 years ago, Apr 17, 2022 01:27 UTC

  • Update

    Heroku Security Update: GitHub integration mitigation steps

    To mitigate impact from potentially compromised OAuth tokens, we will revoke over the next several hours all existing tokens from the Heroku GitHub integration. We are also preventing new OAuth tokens from being created until further notice. Your GitHub repositories will not be affected in any way by this action.

    Currently running Heroku applications will not be affected, but this will prevent you from deploying your apps from GitHub through the dashboard or via automation. Some other actions in the dashboard will no longer work due to this mitigation, and you will be unable to reconnect to GitHub even though you may see warning banners about reconnecting. As a temporary workaround, you can use one of the other code deployment methods available in the following documentation:

    Posted 3 years ago, Apr 16, 2022 04:04 UTC

  • Update

    At Salesforce, we understand that the confidentiality, integrity, and availability of your data are vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an incident we're actively investigating that may lead to unauthorized access to your GitHub repositories connected to Heroku.

    On April 13, 2022, Salesforce Security was notified by GitHub that a subset of Heroku’s GitHub private repositories, including some source code, was downloaded by a threat actor on April 9, 2022. Based on Salesforce’s initial investigation, it appears that unauthorized access to Heroku's GitHub account was the result of a compromised OAuth token. Salesforce immediately disabled the compromised user’s OAuth tokens and disabled the compromised user’s GitHub account. Additionally, GitHub reported that the threat actor was enumerating GitHub customer accounts using OAuth tokens issued to Heroku’s OAuth integration dashboard hosted on GitHub. Based on the information GitHub shared with us, we are investigating how the threat actor gained access to customer OAuth tokens. The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts. With the access to customer OAuth tokens, the threat actor may have read and write access to customer GitHub repositories connected to Heroku. Given the incident is still active, please review the recommended actions provided below.

    What action do I need to take?

    Please follow the instructions below to conduct a comprehensive review of your GitHub logs for evidence of exfiltration. If you identify suspicious activity, please contact security@salesforce.com to share your findings which may assist in our investigation.

    To request logs from GitHub:

    For more information regarding GitHub’s investigation and actions you should take when reviewing your logs, please visit the GitHub blog.

    Additionally, we recommend disconnecting Heroku from your GitHub repositories. Please note that disconnecting will prevent you from deploying your apps using the Heroku dashboard and pipelines will be partially functional. While disconnected, you can use one of the other code deployment methods available in the following documentation:

    If you see evidence of exfiltration in your logs, you should look carefully through your repositories for any credentials that may have been compromised and mitigate access by disabling accounts and rotating credentials as needed. We also recommend revoking or rotating any exposed credentials. Source code credential scanning is one of the primary ways that malicious actors can escalate their access.

    What are the next steps for Salesforce?

    Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers’ OAuth tokens, we will immediately take appropriate actions.

    How can I get more information?

    If you see any evidence of impact, please open a case with Heroku Support.

    Updates will be posted to status.heroku.com as additional information becomes available. If Salesforce becomes aware of unauthorized access to customer GitHub repositories connected to Heroku, we will notify affected customers by email without undue delay.

    Posted 3 years ago, Apr 15, 2022 23:36 UTC

  • Issue

    At Salesforce, trust is our number one value. We're actively investigating a report received on April 13, 2022, from GitHub that a subset of Heroku’s GitHub private repositories, including some source code, were downloaded by a threat actor on April 9, 2022. We proactively notified our Heroku customers regarding this issue and will continue to provide updates to assist them as the investigation continues. If Salesforce or GitHub becomes aware of unauthorized access to customer GitHub repositories connected to Heroku, we will notify affected customers by email without undue delay.

    Posted 3 years ago, Apr 15, 2022 22:32 UTC

Current status