More or less back - what happened to our server the other day
Archive - Originally posted on "The Horse's Mouth" - 2013-06-14 21:48:01 - Graham Ellis
I woke Wednesday morning to an email informing me that one of our two dedicated root servers had been identified as being involved in a Distributed Denial of Service attack and that the hosting centre where it's based (and the company from whom we lease it) would be taking the system offline (said "had", but it was still there) and required us to fix it. A really useful message, as it didn't give me any details what so ever - what was said to be being attacked, or in what way. The message did tell me how to log in to my server through ssh as that was being left available to me. Yeah - that bit's teaching my Grandmother to suck eggs! But then it stopped at that point.
A phone call to our hosting centre was less than helpful; no more information was available, "it's involved in a denial of service attack, but we don't know on what, nor which service is being attacked". OK - I had to ask how they were sure it was involved in DDoS if they didn't know what, where or how. "No more information is available to us". (It has struck me later that there might be some sort of state secret stuff that was being targeted, that our provider was being bullied, or that our provider was simply using the DDoS excuse to force the upgrade of an admittedly very old system). A promise to email me further to tell me more and the "more" was that they had proceeded and taken the system offline. Nice - NOT - and no more information
So, the last 48 hours have been spend reloading / rebuilding sites onto another server. 500 miles from the original one. Different operating system. Different version of MySQL. Different version of PHP. Different web site structures. The DNS redirects took a while to click in, so I got a couple of reports from Google about sites being down. The First Great Western Coffee Shop Forum has many active users; and I was getting alerts from some of them "yes, I know..." and providing feedback. And quiet but important, the Well House Manor site also had to be restored.
I think I'm almost there now: the Coffee Shop's SQL software has stonger password encryption so everyone has to reset their password via an email reminder - a problem for those who have not kept their profiles up to date and accurate, and there are a handful of subtle issues that you're bound to get when replacing software with other software that's several major releases newer. Lots more little things to do, but we're flying again!
Oh - did I mention teaching a Lua course on Wednesday and Thursday, attending a meeting of the First Bus Customer panel on Wednedsay evening, a phone call in early hours of Thursday to tell me that I was on breakfasts from 06:30 (change of plans there) and a long day today with me single-manning the hotel from 6 a.m. to 10 p.m. as 3 sets of holidays and 1 sickness all coincided. Oh - and a day of PHP training to deliver too.
It's coming up to 10 p.m. - I'm still at work and will be delighted to pass the hotel to Heather and Poppy tomorrow. I love the challenge and being able to rise to it and the reward is great. A review of today's course - "Excellent, very useful. Lots of techniques learned". And a forum post "With many thanks to grahame for his efforts in restoring our ability to log into the Coffee Shop forum" ... and another "And I for one am grateful to Grahame for his guidance - I am probably the original nightmare subscriber when it comes to technology ..." as I provide support on getting people running again. I'm going to sleep well, but happy at the work done. Sorry, Heather - I didn't get a chance to change bedroom 1...