Resolved
A reason for outage summary has been posted.
Issue
Outage Date: October 12, 2010 Outage Time: 7:31 - 8:47 UTC
Impact:
The following services were unavailable for approximately 71 minutes:
Root Cause: An internal database server was experiencing severe performance problems with one of it's disk drives. We detected the problem quickly and decided to attempt a restart of the database server. Upon restart, the problematic disk was still unresponsive and we escalated to our hardware provider for further assistance. After receiving guidance from our hardware provider we restarted the machine again it came up normally.
Resolution: Our internal database was eventually restarted and all services were resumed.
Remediation: We have a project underway that will make failing over to a replacement database server much faster. We expect this project to be complete within the next few weeks.
Resolved
Everything is back to normal. We will post a follow up tomorrow.
Update
We are escalating a hardware issue to our provider.
Update
We are restarting a misbehaving database and it's taking longer than expected, We are bringing in more engineers to assist.
Issue
Service will be temporarily interrupted due to an unplanned maintenance operation.