Undergoing unplanned maintenance

Apps 1 hour, 2 minutes
Data 22 minutes
Tools 1 hour, 6 minutes

Activity

  • Resolved

    A reason for outage summary has been posted.

    Posted 14 years ago, Oct 13, 2010 14:35 UTC

  • Issue

    Reason For Outage Statement

    Outage Date: October 12, 2010 Outage Time: 7:31 - 8:47 UTC

    Impact:

    The following services were unavailable for approximately 71 minutes:

    • Heroku command line client
    • Heroku API
    • Web console
    • Application Unidling
    • git push

    Root Cause: An internal database server was experiencing severe performance problems with one of it's disk drives. We detected the problem quickly and decided to attempt a restart of the database server. Upon restart, the problematic disk was still unresponsive and we escalated to our hardware provider for further assistance. After receiving guidance from our hardware provider we restarted the machine again it came up normally.

    Resolution: Our internal database was eventually restarted and all services were resumed.

    Remediation: We have a project underway that will make failing over to a replacement database server much faster. We expect this project to be complete within the next few weeks.

    Posted 14 years ago, Oct 13, 2010 14:32 UTC

  • Resolved

    Everything is back to normal. We will post a follow up tomorrow.

    Posted 14 years ago, Oct 12, 2010 08:47 UTC

  • Update

    We are escalating a hardware issue to our provider.

    Posted 14 years ago, Oct 12, 2010 08:19 UTC

  • Update

    We are restarting a misbehaving database and it's taking longer than expected, We are bringing in more engineers to assist.

    Posted 14 years ago, Oct 12, 2010 07:59 UTC

  • Issue

    Service will be temporarily interrupted due to an unplanned maintenance operation.

    Posted 14 years ago, Oct 12, 2010 07:44 UTC

Current status