Downtown Los Angeles was hit by another power outage last Friday. This affected DreamHost’s services and other tenants at the Garland Building (here among MySpace), because the UPS system failed.
DreamHost was affected by a similar incident last year, when the generators didn’t work.
DreamHost founder Dallas explains:
The power supply for the entire building (which houses several data centers, several web hosts, and some notable large sites like myspace.com) is completely redundant with incredible amounts of UPS power and 5 huge diesel generators. On paper it sounds very good.
During the power outage last year the UPS system worked but the generators did not. That was very bad but we decided to give them another chance. We have a lot invested in our current setup and changing locations is a very big undertaking. The power problems are affecting all of downtown Los Angeles so we would have to move to a completely other city. Many of our senior technical people live near our current office and data center and they would all end up moving themselves or enduring painful commutes in LA traffic. These are not excuses.. just some insight into the reality of the situation for us.
We have been with this building for several years and at the time of the first power outage that was the first time they had failed us. In hindsight, we should have had less faith in them. That’s very easy to see now.
For this second power outage of the local grid, the building’s UPS system failed but the generators did work. 0 for 2. Our building had been on generator power for the entire week since the local power grid had gone out and that was working fine until some sort of electrical malfunction took the generator system down yesterday afternoon. They quickly switched us back to grid power which is how things are now.
In a network like ours, it’s not just a matter of ‘rebooting a server’ as some of you have indicated. We have all user data stored on network file servers for redundancy and to optimize usage, for instance. After yesterday’s power blip all of the 68 file servers but one came up properly. That one that did not resulted in some websites being down for an additional hour and a half or so while we diagnosed the problem. We have now corrected the configuration problem and expect all file servers to come up normally in the future.
Obviously, loss of power to our data center is something we would rather never see happen. We are already in the process of acquiring some UPS equipment of our own for our most critical equipment and we will be evaluating all possible options including completely moving our data center operation to another location.