Overview
We generate statistics for usage of our two frontend web servers. They are avaialble here. We use AWStats to generate the HTML statistics pages from our apache logs. We also have a shell script and a perl script that deal with club-specific issues merging together the logs from all of our frontend servers.
Scripts
awstats.pl
We just use the stock awstats.pl script from Debian.
logresolvemerge.pl
This script comes from the AWStats package. Again, we use the unmodified version from Debian. It is used to combine the apache logs from each of the frontends. A simple "cat" is not sufficient, since awstats.pl requires the log file it reads to be approximately chronologically sorted.
update.sh
This script is only used on the "master" machine.
This is the script run from cron. It is runs as the awstats user, currently out of /etc/crontab, but it probably should run out of awstats' crontab instead. It's responsible for combining the logs from all of the frontends, running the generate.pl script to make HTML pages, and copying the generated HTML to the other frontends. This script is configurable, via /var/awstats/config.
generate.pl
This script is responsible for invoking awstats.pl in the correct magic ways to generate all the statistics we want. It is run by update.sh.
It takes two commandline arguments: an AWStats configuration, and an output directory.
Configuration
/etc/awstats.*.conf
These are standard AWStats configuration files. We have multiple configurations, currently one for each of the virtual servers we are generating statistics for separately, and one for everything.
All of these files include /etc/awstats.default.inc, and override necessary settings.
/var/awstats/data/clusterinfo.txt
This file assigns names to the magic numbers that are discussed below, in the Apache Log Formatting section.
/var/awstats/config
This is just a shell script fragment that is sourced by update.sh. It should define three variables:
- me - the frontend the update.sh script is executing on
- slaves - a list of the other frontends
- configs - a list of awstats configurations to generate statistics for
/var/awstats/domains/*
These files contain a list of the possible names that a virtual host may have. They are referenced in awstats.*.conf.
Other
Pipes
Make sure a fifo for $me, and each entry of $slaves, as defined in /var/awstats/config, exists in /var/awstats/pipes.
Apache Log Formatting
Since we do not have separate access logs for each virtual host, we add the vhost that was accessed in our access logs.
We also add a magic number, which is used to identify which front end handled the request. Currently, we've used the following numbers for the following machines.
- 1 - ??? (one of antimony or arsenic)
- 2 - ??? (one of antimony or arsenic)
- 3 - astatine (current awstats master)
- 4 - polonium (old machine)
- 5 - polonium
- 6 - www2
- 7 - contrib-cgi
We use the following LogFormat directive in our apache configuration.
LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O %V <MAGIC-NUMBER>" combinedVc