3D graphics - web site usage - simple matplotlib and python example
Archive - Originally posted on "The Horse's Mouth" - 2010-10-12 22:22:13 - Graham EllisSome very interesting graphs from our server log data, courtesy of Python, numpy and matplotlib. Truly, a picture paints a thousand words. The data in the first and last diagrams is raw - showing exact number of hits per hour; in other diagrams I have used proximity smoothing which makes the trends very much easier to spot amongst lessened noise, but eliminates a peak late in the evening of day 10 which I am going away to investigate when I have posted ;-)
The source code of this example is [here] - the data is unpublished, as it comprises some 24 files of between 20mb and 44 Mb each - an illustration of how powerful a Python tool can be.
Update ... added the next morning I've looked at that late evening spike on 28th September, and it turned out to be some bloke (or blokes) from Almere in Netherlands who looked at just under 20 pages on our site and decided we were so good he would mirror the rest. An automated program, but one claiming to be IE5, was used. He did check robots.txt - but he went for as many pages as he could as fast as he could and caused the spike which caused a noticeable blip on the graphic, with 4416 requests between 10 and 11 pm when the site would otherwise have been quite quiet. The shape of the spike is further explained by 2419 requests in the following hour.
I doubt whether this user would have caused a problem to others even if he had gone "full tilt" as he would probably have been limited by his bandwidth, but it was a good test of our "governor value" which duely kicked into action.