Sep 23 2008
Don’t put too many eggs in one server
Last night I came home from a 4 week road trip with my wife and daugther through Europe. Since my wife doesn’t have a license, I alone drove over 7000 kilometers. It really was an incredible trip though. We’ve seen alot of nice things.
I also managed not to think about work for a while there. But suddenly - probably due to feeling a disturbance in the SEO force - I felt a sudden need to check how my servers were doing only to find out that one of them was down. I checked some Analytics stats and it had been out for 1½ day. I tried to restart the server using my cell phone but the control panel had implemented some fancy AJAX thing for that particular control and the Opera Mobile Browser couldn’t cope with that. And when abroad cell phone traffic is crazy expensive. It’s like 10$/Mb or something - hate the networks for that.
Finally I called the support to have them restart the server for me.
Alltogether a downtime of three days. That’s not good at all and the really sad thing is that my most important projects are on that server.
What can I learn from this?
- Monitor my servers better, ie send mail/sms
- Spread the risks, not having large servers but several smaller ones
- Have plans made out to deal with these sort of things
It shall not happen again, I promise.
September 23rd, 2008 at 10:25 pm
Shit happens unfortunately, many times there are not much we can do about it. I’ve had problems with servers too and know how frustrating it is. I can only imagine how it must be like to not be able to do much about it and being far away. I think the expression “don’t put all eggs in one basket” is applicable to anything, hosting included but it is also convenient to run as much as possible from one place. I guess it must be a calculated risk. As with all things that can go wrong they eventually do, and as long as it hasn’t happened to you yet you think it only happens to others.
September 25th, 2008 at 4:37 pm
It’s always better to spread risk but you have to weigh in cost, convenience and value as well. How big is the risk, i.e what’s the likelihood that the server goes down, how much do you trust the host? Is it critical, i.e what does it cost you for n hours downtime?
I know this was a serious blow and am sorry it happened to you. If you or anyone else have any tips on how to avoid things like this or how to reduce loss if it happens please share.
December 22nd, 2008 at 11:10 pm
Additional servers also create their own problems such as patches, updates, settings, etc. No easy solution.
It does seem ridiculous to not support mobile devices for reboots.