Power Outage Update - 02/July/2009 10:49AM

 Dear customers!

At approximately 10:49AM AEST 02/July/2009 we lost power to a rack which contained our primary site, and a number of other servers.  This resulted in an outage extending until 11:01AM in which power was restored, and servers slowly came back online after this. Some servers may have taken longer to come online due to forced filesystem checks, or manual intervention required.

Upon investigation it was concluded that a dip in the incoming utility power to the facitily caused one of the UPS systems to move from utility power to backup power, as expected.

At this point in time the UPS sytem was impacted by a faulty time controller, causing the UPS to fail. Standard operation saw a load transfer from the A3 UPS to the redundant B UPS system. Due to the faulty timing controller, an out of phase transfer via the Automatic Transfer Switches (ASTS) simulated a high voltage in-rush condition, which caused the B UPS system to decrease load, resulting in the loss of both primary and redundant power services to equipment provisioned on the A3 power system.

This failure scenario occurred in too short a time frame for generator support, with the utility power dip resolving within 2 seconds to affected UPS systems.  The faulty timing controller in the UPS system has since been replaced.

Please accept our sincere apologies for any inconvenience caused due to this outage, and be rest assured we are investigating several mechanisms to prevent this sort of outage from happening again in future. 

If you had an outage greater than 40 minutes, you may be eligible for SLA service credit.  To claim SLA credit, please call us on 1300 66 2 100.  Be sure to include when the outage started, when it finished, the IP address or domain name of your account, and any relevant support tickets.  If you do not provide this information, you will not be provided with SLA credit.

Do not hesitate to contact us if you have any questions or concerns.