Identifying and clearing denial of service attacks on your Apache server
Archive - Originally posted on "The Horse's Mouth" - 2014-09-27 09:52:42 - Graham Ellis
If ... ..... .... I ..... ..... were ...... ... ... ... to . ...... . write .... .... .... . a ... ...... ... ..... sentence, .. ... but ...... .. drip ...... . ..... ...... ... the ..... ... .. ...... ...... words . . .. out ..... .. ...... ..... . slowly ..... ... ...... with .... ... ..... ... long ... . ... ..... pauses ...... . . .... ... between ..... them, .... ...... I ...... ..... ..... ...... could .... .. ... .... burn ... .... up .. . . ...... a ... ..... .. lot ..... ...... ...... of ..... ... ...... . ..... your ..... .. time, . dear ..... .. .. reader, . .... ..... as ...... .... ...... ... you .... . .. parsed .. .. ... ..... ..... through .... ...... . ...... what ..... .... I'm .. .. ..... saying . ... ... and .... .. come ...... ..... ...... ... ... to ...... the .... the . ..... . end ..... ..... .. ..... .... sentence. .... ...... ....
And if I was to ask you a question. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then ask it again and again. Then I would end up burning up a lot of your time again.
Although we're not rude and inconsiderate of each other in this way face to face, such inconsiderateness is often shown - be it accidental, unthought, or intentional by browsers and browsing programs when they visit web sites, and web site administrators must make consideration of such activity, which can be ongoing 24 hours a day, 7 days a week, when they make their web sites visible.
The text above generated by this Python program - you didn't really expect me to handcode all that did you! ... See our Python courses where you can learn this (and other more conventional) uses of the language!
Our web servers get accessed from time to time by thoughless people such as I've described above (and by thoughtful people who intentionally do this sort of thng to lots of people, looking for security holes), and we need to keep an eye on our loadings and watch how our servers are doing. An old blog (about 4 years ago) tells you how we do the monitoring and graphing - it's here and the techniques are still current and valid, and a very recent discussion / item on our First Great Western Coffee Shop Forum - [here] - shows you how we located and overcame some issues a couple of weeks ago, looking at server log files, using Perl scripts (described here) to analyse the daily logs and find the needle that's causing the pain in the haystack of valid traffic.
From the last 48 hours, and again during the night just gone, I noticed some noise in the standard pattern that I expect to see from the server graphs ... here is the current [16:00 update] graph as I write:
And I pick out from that:
a) A big peak one evening. Not a problem, as that's the time that server backups were processing; I would prefer the peak to be not quite so high during this procedure, and indeed I introduced a recovery delay at a couple of points during the backup procedure recently, whilst making sure that each distinct web site is backed up without long gaps so that any content changes during the procedure will not cause a problem ("syncronisation").
b) A rising level of traffic yesterday, with the orange line being noticably above the lines for previous days all the way through the evening. Using the Perl scripts linked to above, I was rapidly able to take a look at the log files through a filter and see that one single IP address was requesting our hotel guest book that goes into each room ... and requesting it again and again - in total there were over 56,000 requests at one second intervals. You'll notice today's black curve in the graph above distinctly drops from around 03:00 when I told our server it could stop answering these requests for 2 Mbytes per second!
c) Sudden upward loading spikes - including one after I had fixed (b) at about 07:40 this morning. Taking a look at the server status pages (which are available to me as server admin), I notice a rather curious pattern during the spike of busyness:
The diagram is showing all the current threads accessing the web server. (Please ask if you would like me to teach you about these things, and / or take a look at your server!). And each of those lines marked "reading" is a remote browser that is dripping a question in, ever so slowly and with lots of pauses in between just as the first text I started this article with. Result - everyone else who's making normal traffic requests is having to wait until there's a thread available, and / or lots more threads are opening on the server and the machine's getting rather full.
The solution - something we've done before on another server - is to make our server a bit less patient, and to give up more quickly on requests that are dripping in slowly. The resulting server status list looks more like
which I can assure you is much more like what I expect so be seeing. As a reader of this article, you might not appreciate just what is and isn't right for these diagrams - they're things to look and and learn on your server, and learn about Python and / or Perl too so that you can do the extra analyses to look for patterns when things aren't quite as you would expect ... and do so soon, rather than waiting to when you have problems to resolve and can't take a dynamic look at the "when it was working" case.
