Backup procedures - via backup server
Archive - Originally posted on "The Horse's Mouth" - 2015-01-24 15:15:16 - Graham EllisOur main web server system becomes ever more important to our business - and having procedures in place to restore systems in the event of failure becomes ever more critical.
What are we protecting against? Both hardware and software failures, and against intrusions and obsolescence (to the extent of being unuable) too. And these come in different flavours - a hardware failure (in the shape of a power supply blowing) was quickly fixed by moving the disc into a new chassis / onto a new processor, whereas a software failure that's done laten damage a time back and just surfaced can take considerable fixing!
We have three phases of backups
Phase 1
a) On server, our most dynamic of databases are backed up every hour, and our less changing databases every six hours
b) On server, our whole web site and database areas, and our /usr/local, are backed up twice a week
c) On server, a monthly backup takes in just about everything thatcan change
Phase 2
a) Our every-six-hour backup gets pulled, twice a day, onto another server (phyically in a different country)
b) Our twice a week backup similarly gets pulled onto that other server
Phase 3
From time to time, we grab copies of the backups to our own inhouse server and / or laptops
We have stopped short of transaction logging, mirror servers, etc; almost all of our server content is pretty static and the mos dynamic - the First Great Western Passenger Forum - is a free forum without income, and users shouldn't (and don't) expect robustness an order of magnitude better than the services they talk about. We lost a couple of posts the other week, and the server was down for an hour or two. Course descriptions / Well House web site hanges are uploaded from our developemnt systems for the most part, so if we loose anything on the main server we can just replace it.
Phases 1 and 2 are performed by regular time jobs under crontab - with phase 1 being shell scripts (see [here] for hourly, and [here] for twice weekly
The Phase 2 script are programmed in Expect - an easy solution to grabbing files over the network, mimicing the command line rather than some of the complexities of setting up keys and keychains. Yes, I know it's less secure, but then virtualy of all the data is public stuff anyway. The expect script is [here].
The ultimate question with all of these peocedures is "if the live system fails at the worst possible time, and in the wort possible way, how do we get it back, how long does that take, and what is the cost - financial, to the business, and in time. And those question should be the starting point as you work out your backup strategy.