Main Content

Backups by crossover between network centres - setting up automatic scp transfers

Archive - Originally posted on "The Horse's Mouth" - 2013-04-13 20:02:56 - Graham Ellis

Our web servers don't need transaction logging backup systems - emails and secure pages are looked after (and spam trapped) by a separate machine from our two dedicated web servers, which handle a great deal of requests for data reads, but data writes aren't all that many. But of course the data writes / changes / blog articles / forum contribution / calendar changes are happening all the time, and there's a need to be able to restore the machines if something goes dramatically wrong. The question that need to be asked in setting up the backup strategy is "how do we get this back / how much do we loose if the nastiest thing goes wrong at the worst moment?"

Firstly, both machines run regular and quite frequent backups using crontab jobs - with the backups being stored on the same machine, so that we can pull back any data that we need to. Most of these are stamped with the time in the week at which they're taken, so we can step back up to 7 days. These backups have proven very useful when members of forums have done something very silly (like decided to leave in a huff and delete all their posts) ... and we've restored the posts - and thus the integrity of the threads - very quickly indeed. Since of backups are at an SQL and file level, such a restoration doesn't even mean any loss of recent data since the last backup was taken.

But - and this happened a couple of weeks ago - what happens if we have a hard disc failure? "Sorry - there's no point in trying to mount the old disc - there's nothing left on it that can be read...". That's were offsite backups come in and, until not, we've taken such network copies about once a week, when we remember. OK - that's not been very clever, and it takes time - so I've now put into place a data swap scheme between our two machines - on in Germany and the other in England. The question arose how to transfer the backup files cleanly, automatically and securely, and Ive set up a crontab job, twice a week each way, using scp in batch mode. And in order to do that, I had to set up public and private keys between the backup accounts on the two machines. The setup only needs to be done in one direction, as the authorised client scp can both push and pull files. Here are the details ... with keys intentionally changed so that no-one reading this blog can get in!

On the WELL HOUSE MANOR machine - which will be the client that runs the scp command - in Germany

  -bash-3.14b$ ssh-keygen -t dsa
  Generating public/private dsa key pair.
  Enter file in which to save the key (/home/backup/.ssh/id_dsa): my_client_key
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in my_client_key.
  Your public key has been saved in my_client_key.pub.
  The key fingerprint is:
  21:17:81:55:4d:87:aa:e5:0f:34:75:d7:6f:d4:fe:ca backup@p15161732.pureserver.info
  -bash-3.14b$ scp my_client_key.pub backup@www.wellho.net:
  backup@www.wellho.net's password:
  backup@www.wellho.net's password:
  my_client_key.pub 100% |**************************************| 623 00:00
  -bash-3.14b$

On the WELL HOUSE CONSULTANTS machine - which will be the server - in England

  -bash-4.1$ cat my_client_key.pub >> ~/.ssh/authorized_keys
  -bash-4.1$ chmod 600 !$
  chmod 600 ~/.ssh/authorized_keys
  -bash-4.1$ rm my_client_key.pub

On the client machine in Germany - copy command in the crontab file for user "backup"

  45 7 * * 3 scp -B -r -i /home/backup/my_client_key backup@www.wellho.net:Wed /home/backup/remote
  45 8 * * 3 scp -B -r -i /home/backup/my_client_key /home/backup/local backup@www.wellho.net:remote
repeated TWICE a week, different sources and targets, in case a system fails during transfer

I'm still going to be downloading the occasional full backup to our own HQ and hotel, and also to that mail and secure server I mentioned (which are in California). However, if the world's hit by a disaster big enough to wipe of data centres in London, Koblenz and Fremont, I expect that Lisa and wouldn't be doing very much IT training thereafter!