« Making Thunderbird fly | Main | The Sorry Sony Saga »
November 11, 2005
Battered by e-mail gods
Wasn't it just a day or so ago that Bleeding Edge put our hand on our heart and declared how happy we were with FastMail? We should have known better. We haven't been able to access our email for most of the day, due to the fact that our account happens to be on a server that initially had a failing hard disk, and now appears to be in the process of being replaced.
Now look, Jeremy. If you're going to have a hard disk failure, kindly have it on a server that DOESN'T carry our account! OK?
And we know that you've probably got your hands full right now, but we'd love to get a report from the horse's mouth. What's been going on?
Posted by cw at November 11, 2005 06:03 PM
Comments
Backing up a primary email server and then having to restore it in a timely fashion is extremely difficult.
So, the other great Australian-owned email service AlienCamel has implemented a duplicate email server on operated on another continent that houses all our users' email for the past 24 hours. It's real-time mirroring so if our primary service is unavailable for whatever reason the emails will be available on the hot-duplicate server - immediately.
Posted by: syd of aliencamel at November 11, 2005 07:21 PM
You can see the latest on the Status Blog. Only 5% of users are impacted by this - unfortunately Charles, that includes both you and me!
It's the first time we've had a disk failure cause an outage. Our disk units are designed to handle 2 disk failures. Unfortunately, we had 3 disks in the same unit (which holds 12 disks in total) fail within an hour of each other! We can't see why this happened - the temperature and power logs in our data centre don't show anything unusual.
Posted by: Jeremy Howard at November 11, 2005 07:33 PM
Actually, it's been "one of those days". Right in the middle of the hard disk disaster, the toilet door at our office broke and I got locked in the bathroom. Rob had to literally break the door down! It took him about 10 tries - it's not nearly as easy as they make out on Law & Order...
Posted by: Jeremy Howard at November 11, 2005 08:27 PM
Really it was more bizarre than anything else. 3 computer geeks standing round for 30 minutes trying to work out how to open a door. The bathroom is on the second floor, no window, hinges on the inside, door lock is the type you can't use a card or the like to slide past it to open it. We tried unscrewing everything we could (didn't help us actually undo the locking part), tried getting a screwdriver in the room (just couldn't quite get under the door), tried to get something in to knock the pins out of the hinges (nothing really pointy enough around, and you'd need a hammer inside anyway, we tried in another bathroom with a thin pen to test it by using a drawer from the cabinet as a hitting weight, but not enough leverage)...
After 30 minutes of intellectual attempts, it came time to try and break the handle and lock (no good, even jumping on the handle from a chair wouldn't break it) so finally it was just knock the door down.
How bizarre....
Posted by: Rob Mueller at November 11, 2005 10:22 PM
Wow - sounds like your landlord has breached some health and safety regulations! Toilet locks should be able to be opened from the outside in case of emergencies??? :)
Posted by: syd of aliencamel at November 12, 2005 09:10 AM
Yes Syd, it can be opened from the outside. However, the mechanism failed, so neither side worked.
Posted by: Jeremy Howard at November 12, 2005 10:21 AM
My commiserations JH. If you are mentally recovered from the toilet breakout and the email disaster, one weird slant I can share is that several people I know point to a coincidence and then imply to me that incident A caused incident B, even though technically, logically, physically, they are not related at all. Said friends on reviewing the facts of your case would say - "Ah, you abused the mail server by forcing it to push out 15Mb emails, you broke it" - to which I would typically say - "no such thing". Welcome aboard. 8-)
Posted by: anandasim at November 12, 2005 10:29 AM

