Main Content

Server logs - drawing a graph of gathered data

Archive - Originally posted on "The Horse's Mouth" - 2010-11-03 05:41:12 - Graham Ellis

You can log pretty much any server data on a Linux or Unix system at regular intervals using a crontab job that adds a line onto the end of a file. That crontab job could be as simple as a shell command, or it could be a bash or perl script (the two most natural languages to choose for such a task) or something else. As a part of last week's work, I wrote such a logging script (source code [here]) and on yesterday's PHP course, I picked up the data files that I've generated over the last few days and produced this graph of server useage. If I were to show you the raw data file, you would find the pattern that the graph shows almost impossible to spot.

What patterns *can* we spot? Well - there's a distinct event of some sort at around 4 a.m., and there was a bigger one around 5 a.m. one day. I can 'account' for these - we schedule some nightly admin / tidy up work on our server, include a couple of quite slow analyses of our site and logs which result in summary reports of the previous dey being emailed to us. And, twice a week, the system backs itself up; that's the big bump in the yellow curve. Otherwise, I'm going to say "time will tell" with these graphs. I'm tempted to let traffic be logged for several months and add the results up to look for further patterns - we already know from other traffic graphing that the pattern varies significantly based on the day of the week, so I can't just combine all the days I have logged to even out glitches!


When I post this article, the graph shown here will look almost identical to the one above - but as time passes it will vary. For this second display is the "live" one! It's using PHP's gd module - which is shipped with PHP these days, although you need to configure it in. The application behind the graphic looks for the latest five files of log data, and reads them in. Rather than printing out the results of its analysis, it makes calls to functions like ImageFilledRectangle and ImageTTFText and ImageLine which draw into a canvas or pixel map held on the server, and once it's completed that it calls ImageJpeg on that canvas to convert it to a .jpg image which is then sent to the browser for display.

The complete graphic script is available [here]; I cover graphics in PHP very briefly on our PHP Programming course, but if it's something you'll be seriously using you should also look at our PHP Techniques Workshop which is a 2 day intermediate / advanced level course.




Notes:

1. I had a "management choice" as I wrote this example - whether to use a graphing package such as phplot - see [example] or whether to stick with gd. I elected to stick with gd which comes as a standard part of the PHP distribution these days (though it has to be configured in) rather than creating a dependency on an external piece of code if anyone wants to grab a copy of this and try it for themselves.

2. The sample graph I captured at the very top of this article includes the day that the clocks went back ... so there's 25 hours of data showing on one of the lines; there's a potentially long discussion about how the time change should be handled in the application, but in the overall run of things it's not going to make a significant difference, so I have simply let the curve extend. There might also be other glitches if we have to adjust the server clock, or if the server goes down, and we may loose a sample if it's rebooted.

3. Images shown on this page are scaled down. Full size, current, image [here]