An overpractical test of our backup strategy!
Archive - Originally posted on "The Horse's Mouth" - 2013-03-30 20:11:33 - Graham EllisExercises are all very well - but the real test of how your emergency procedures work are in a real emergency. So it was that at 3 O'Clock this morning I noted something very odd with our server load; logged in and noticed a process called "shred" running a root and got chucked off about 60 seconds later. The server died, responding only to ping.
Cutting a long story short, I'm pretty sure it was some sort of malicious attack on our server (I'm not going to speculate on what it was exactly here) and I found myself filling in tickets and discussing the issue with an engineer through that system who, however, was very good. But rapidly came to the conclusion that a rebuild of the main server was necessary and whilst he could mount the old disc on there too, there was little point as it hadn't even got any partitions.
So at 4 O'Clock I found myself with a new server build. With minimal stuff. No running web server, not even a compiler and CentOS 6 rather than Red Hat Fedora. It's been an interesting day - we teach delegates not to use too many technologies, but then we ourselves use a lot in training and have to have them all live and demonstrable.
Backups were 3 weeks old - not really all that frightening as we can reset from the development server. Fixes in odd places have to be remembered too. Now where did I get THAT library from? And where do I get the equivalent for the current software version? At least we're now at PHP 5.4 and MySQL 5.6 ...
16 hours later, I'm just ending today's sprint. The services the machine offers are up and running again. Most of the main website content is too, and the working scripts. I have a list of issues, 7 other virtual hosting sites and much more and I have just re-backed up 4 Gbytes. Time for some zzz zzz zzz