Gallery

How to make log files cool

Posted on 3 Mar 2009 at 17:26

Ian Wrigley shows you how to spruce up your website stats, and takes a trip down the Amazon Management console.

We've been in a rather fragmentary work mode over the past few months, working on several projects that required the use of various tools and technologies. This month's column examines what tools we've been using as we enter 2009, kicking and screaming. First off, how to make boring log files look cute.

Log files are, as any network admin knows, vital sources of information: an expert can glance through a log and immediately spot the problems, then dash off a script to analyse the log and extract whatever salient pieces of information are required, and finally hand that data to the programmer who's going to fix it. But then there are the silly requests that you have to satisfy. One of these, which seems to be becoming more popular these days, is from the client that wants to "see" how popular their website is in real-time. We recently encountered a client that wanted to place a large-monitor in its reception area, displaying fascinating statistics about the popularity of its site.

Now of course, it would be easy enough to display this site-visit information textually, as merely running the command tail -f on the log file would do the job. "Tail", for those who aren't familiar with Unix, is a utility that displays the last few lines of a file, ten by default, and by applying the -f switch you're telling it to display those last ten lines plus any more lines subsequently added to the file. Try it, it's fun! But that's not at all what was required, because "it should look cool" was the specific request. This is undoubtedly a response to all those movies that feature a big control room full of monitors showing leaping real-time graphs, dots running around maps and so on. The problem lies in deciding just how to visualise in a cool fashion what are, for the most part, fairly pedestrian traffic statistics.

Fortunately, it turns out that there's a very cool piece of software that will do exactly what's required. It's called glTail, by Erlend Simonsen, and it's a Ruby program that provides an extremely funky, yet practical and useful, visualisation of data from just about any log file format. Parsers are built in for Apache, Rails, IIS, Postfix, MySQL and other logs, and you can write your own parser if your log file format isn't natively supported. The glTail display takes the form of a stream of dots or balls flowing from the sides of the window down toward a central repository - the more balls, the more requests; and the larger the ball, the larger the size of the request (for instance, the size of file served up by Apache). The program supports tracking multiple log files simultaneously, even on different servers, and as the documentation says, "if you can 'tail' it, you can visualise it."

Since glTail is written in Ruby, installation is simple: just download the code and install the Ruby Gems it needs, namely net-ssh and ruby-opengl. It works on Linux, Mac OS X and Windows, and the only problem we encountered during set-up is that the configuration instructions are somewhat scanty. Basically, you edit a configuration file, but the only information on the options available has to be gleaned by looking at the sample configuration file (or, of course, by reading the glTail source code).

However, even with that caveat, it didn't take long before we were looking at a beautiful stream of dots showing hits to a couple of the sample sites we tried it on. Note that to run glTail simply requires that you be able to ssh into your remote server to access its log file, with no change to your own server's configuration necessary.