Getting more log information from the Apache http web server
Archive - Originally posted on "The Horse's Mouth" - 2011-09-16 09:43:23 - Graham EllisThe Apache httpd server is preconfigured when you download it to define common and combined logging formats - and those are the standards accepted by so many tools. But there's so much more information available if you choose to log it. In order to get a better understanding of metrics of the traffic on one of our dedicated servers for a few days, I've added the following log lines to my httpd.conf:
LogFormat "%a %v %{%Y%m%d.%H%M%S}t %s %>s %B %T %P %X \"%{Accept-language}i\" \
\"%{Accept-encoding}i\" \"%r\" \"%{If-modified-since}i\" \"%{Referer}i\" \
\"%{User-agent}i\"" extras
CustomLog "/home/wellho/logs/extra_log" extrasHere are what those fields call for:
| %a | remote host ip |
| %v | Your virtual host serving the request |
| %{%Y%m%d.%H%M%S}t | The time the request was made (In our own specified time format) |
| %s | The returned Status from the request |
| %>s | The returned Status in the response |
| %B | return size (0 for no content) |
| %T | Time taken to serve the request (seconds) |
| %P | The child process ID that served the request |
| %X | Completion Status |
| %{Accept-language}i | Language preference of user |
| %{Accept-encoding}i | Compressions accepted by user |
| %r | The request that was made |
| %{If-modified-since}i | Whether request was time stamp conditional |
| %{Referer}i | Page from which this request was called |
| %{User-agent}i | The type of browser that's making the request |
There's more available too - see http://httpd.apache.org/docs/current/mod/mod_log_config.html
Here are some examples:
203.99.217.10 www.wellho.net 20110916.085049 304 304 0 0 10541 + "en-us,en-securid" "gzip, deflate" "GET /commonimages/f_logo.jpg HTTP/1.1" "Sun, 20 Mar 2011 17:47:06 GMT" "http://www.wellho.net/regex/javare.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8)"- Microsoft Internet Explorer 7, Under Windows XP ... requesting an image only if it had changed (which it hadn't). Called up from a page on Java Regular Expressions, and the request took less that a second to serve. Requester prefers answer in US English, and will accept compressed content that's gzipped or deflated.
95.108.240.250 www.wellho.net 20110916.085314 200 200 116845 0 10651 + "ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01" "gzip,deflate" "GET /forum/The-Tcl-programming-language/Re-expect-script-aborting-in-the-middle.html HTTP/1.1" "-" "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"- A request from the Yandex Spider for a forum page. Prefers a response in Russian (?) ...
27.107.186.224 www.wellho.net 20110916.085921 200 200 12344 18 11098 + "en-US,en;q=0.8" "gzip,deflate,sdch" "GET /bing/images/WHClogoBW.jpg HTTP/1.1" "-" "http://www.wellho.net/net/search.php4?search=use+of+structures" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.0 Safari/535.2"- User running the Chrome browser on Windows Vista, calling for an image. For some reason (to be further investigated) it took 18 seconds for our server to send out the response.
180.76.5.136 twcrp.org.uk 20110916.090108 200 200 2320 0 11132 - "tr-TR" "gzip" "GET /community/index.php?action=printpage;topic=79.0 HTTP/1.1" "Tue, 16 Aug 2011 16:23:28 GMT" "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"- Another Spider request - this time with a preference for a response in Turkish. The page requested was from the twcrp.org.uk web site and not from the www.wellho.net site - our server looks after multiple domains. And the "-" 9th field indicates that the server is not to keep the thread alive when the request has been completed; you'll see a "+" on other sample records indicating that the thread is to be kept alive for a few seconds for follow up requests; an "X" would indicate a request that wasn't completed because the browser went away.
77.88.27.25 www.wellho.net 20110916.090659 200 404 16210 0 11844 + "ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01" "gzip,deflate" "GET /dcms/index.php HTTP/1.1" "-" "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"- This final example shows a page that was initially anticipated to be a good request (i.e. was going to send back a status 200), but our server took a further look at the request and decided to send a "page not found" error back instead. We have a number of URL patterns that are rewritten by mod_rewrite which run our user scripts, but respond with code 404. This shows the difference between %s and %>s in the format specification line.
We cover the configuration and deployment of the Apache httpd web server on our Deploying LAMP course.