This page describes, in Unix manual page style, a Perl program available for downloading from this site which allows a system administrator to watch entries, as they are added, in any number of log files on one or more machines on a network. Logs which transfer to new files are automatically followed, and an option allows translation of numeric Internet addresses into the corresponding hostnames where possible. Log items can be relayed to one or more other hosts on the network, permitting a machine's local log files to be monitored there.

NAME

SYNOPSIS

DESCRIPTION

As more and more services have come into use, log files have proliferated. Today, an administrator might want to keep an eye on:

• Master system log
• FTP transfer log
• Web (HTTP) server access log
• Uninterruptible Power Source (UPS) log
• Network Time Protocol synchronisation log
• Automated backup system log
• Firewall suspicious access log
• Logs from local services such as Speak Freely Look Who's Listening and Echo servers

and various others. Many of the widely-used dæmons for these functions do not use the common Unix log mechanism, insisting instead on appending entries to their own private log files. In the case of commercial firewall, backup, and UPS dæmons, source code is not usually available, so modifying them to use the system log is not an option.

When there was a single log file, monitoring it couldn't be simpler--just use:

    tail -f /var/adm/messages

(or whatever the log file is called on your system). The -f option causes tail to start at the end of the file and periodically monitor it for growth, printing any additions to the file. Unfortunately, tail only monitors a single file, so the only way to watch multiple files is to launch separate jobs each running tail on an individual file--yuck. Worse, to prevent log files from growing without bound, consuming all available disc space, many programs that write logs provide a mechanism, automatic or semi-automatic, to periodically cycle the log to a new file, renaming the previous log file. Unfortunately, tail -f does not detect that this has happened, and just waits forever at the end of the old log, not knowing a new file is now being written. If servers are run on a number of different machines (or you simply want to watch Unix log files on all the hosts you're responsible for to keep an eye out for developing disc problems, etc.), you would then have to rsh copies of tail -f on every machine you wanted to monitor, and then make sure the process got restarted when a machine rebooted.

Enter logtail, which allows you to monitor any number of log files on any number of machines (assuming they support Perl with networking and use ASCII log files). Logs from any machine can be echoed to as many other machines as you like. Log files are closed and re-opened at a user-defined interval, so cycling of log files into new files is followed automatically.

OPTIONS

-d

Log messages received from other hosts are normally prefixed with the hostname followed by a colon, for example, ``wombat:''. Specifying the -d option causes the full domain name to be shown: ``wombat.marsupials.au:''. If host name lookup on your machine does not return full domain names for local hosts, this option will have no effect.

-ehostname

Messages from logfiles on this host are relayed to a copy of logtail running on machine hostname. If you want to only relay messages, and not also print them locally, specify the -q (Quiet) option as well. Messages are sent to the named host on the port given by the -t option, defaulting to port 5741 if no -t option is present. If no copy of logtail is running on hostname and listening on the proper port, relayed messages will silently go to packet heaven with no indication they're being lost. You can relay to as many other machines as you wish by specifying multiple -e options; the same port is used for relays to all hosts.

-l[port]

Listen for messages relayed from other hosts on the specified port, or port 5741 if no port is given. Note that you must specify the -l option in order to receive messages relayed from other hosts; otherwise logtail only examines local log files.

-n

When the -r option is specified, logtail normally uses fork to spawn a child process to perform each host name lookup. This blocks processing of other items until the possibly lengthy host name lookup is complete. The very act of creating a large number of child processes may itself, however, cause problems on certain implementations of Unix. In particular, some systems leak resources when processes are created and destroyed which, in time, may bring the system to its knees. The -n option, specified along with the -r option, causes logtail to perform host lookups in the main process instead of spawning a child process for each. While this may serve as a work-around on lightly loaded systems, the forced serialisation of lookups and log item processing will generally prove intolerable on servers with moderate to heavy load. Conversely, if your accesses are exclusively from local machines whose names can be looked up rapidly, the -n option may reduce overhead by avoiding the creation of processes for lookups which can be completed without significant delay.

-pinterval

To avoid filling up disc drives with huge, ever-growing log files, many dæmons periodically, either automatically or at the direction of a user, cycle their log files; the existing log file is closed and renamed, logging continues in a newly created file, and the oldest log file is deleted. When a log file is cycled, logtail has no way to directly discover this has occurred. If it continues to watch the old file, it will not find entries written into its replacement. To mitigate this situation, logtail periodically closes and re-opens the files it monitors. This allows logtail to begin monitoring the new file instead of the now-renamed older cycle. By default logtail performs this close and reopen operation every 15 minutes. To change the frequency, specify the -pinterval option with interval the desired time in minutes. Specifying an interval of 0 completely disables the close and reopen mechanism. If you know none of the files being monitored are ever cycled, this will eliminate the overhead of periodically closing and reopening them.

-q

Quiet: don't print log items on the local machine. If you want to relay log items to another host or hosts with the -e option but not print them locally, specify the -q option. This makes sense for "headless servers" administered from other workstations. You can specify the -q option without the -e option, but doing so simply consumes system resources while producing no output whatsoever.

-rtime

When the -r option is specified, logtail attempts to replace any sequences which resemble numeric Internet addresses (IP addresses) with the corresponding host name, looked up with gethostbyaddr. For example, an HTTP log item such as:

172.27.151.132 - - [05/Nov/1997:17:05:12 +0100] "GET / HTTP/1.0" 200 2668

might be translated into:

roswell.area51.af.mil - - [05/Nov/1997:17:05:12 +0100] "GET / HTTP/1.0" 200 2668

If the host name lookup is unsuccessful, the original numeric address remains intact.

Host names corresponding to numeric IP addresses are saved by logtail in an internal cache so multiple accesses from a given host do not require repeatedly querying the system's host name lookup mechanism. Entries in the name cache are purged after time minutes of inactivity, with 10 minutes the default if no time is given.

At first glance, this host name lookup may appear to be the niftiest single feature of logtail, but before you turn it on, ponder carefully the consequences for the system running the program. If, for example, you're monitoring the HTTP access log of a heavily loaded Web site, the -r option may result in thousands, perhaps millions, of calls to gethostbyaddr every hour, most of which will require multiple Internet accesses to resolve the host name from the IP address. In order to make these requests without blocking the processing of other items, logtail performs each of these lookups in a separate Unix process, but if your system's resolver is single-threaded, everything may still grind to a halt due to the resolver bottleneck. Worse, your own resolver and those of upstream Internet sites may grow to enormous size due to all the requests cluttering up its own internal cache.

As long as the volume of log items is relatively small (say, a couple per minute), and/or the vast majority of references are to local hosts whose names can be looked up almost instantaneously, the -r option can substantially increase the comprehensibility of the log. But if you switch it on while monitoring a million-hit-per-day Web site, you're almost certain to end up regretting it.

-stime

logtail will sleep time seconds between checks for growth of the files it's monitoring. The default, one second, is the same used by the -f option of tail. If the files you're monitoring are only updated infrequently, a longer sleep time can reduce the overhead of running logtail. If you've specified the -l option to listen for messages from other hosts, choose a sleep time appropriate for the fastest-arriving stream of relayed messages. If the sleep time is too long, relayed messages may be lost if the network input queue overflows.

-tport

Transmit messages relayed to other hosts with the -e option on port. Note that the same port is used to relay to all hosts named in -e option specifications. If no -t option is given, the default port of 5741 is used.

-u

Print how-to-call information and a summary of options.

-vlevel

Verbose: print internal debugging information. You probably won't find this interesting unless you're modifying the program. The greater the level, the more detailed (and voluminous) the debugging information will be.

-wcols

Wrap output lines at column cols. By default, lines are wrapped at column 79. A cols specification of 0 disables line wrapping. logtail attempts to wrap lines so as to preserve their meaning. Lines are broken at white space whenever possible. If this cannot be done (frequently the case in HTTP logs which contain lengthy CGI GET requests), an attempt is then made to break the line at a comma or ampersand, major delimiters in such strings. Finally, if the line still cannot be broken, it is split at a slash or plus sign. If none of these delimiters succeeds in breaking the line, it is simply output unbroken. Continuation lines are indented to distinguish them from the first line of the next log item.

BUGS

If you're running logtail on a non-System V Unix (for example, BSD-derived systems such as SunOS 4.x), you may have to change the definitions of $AF_INET and $SOCK_DGRAM to agree with those in your system's /usr/include/sys/socket.h file. Please refer to the comments at the beginning of logtail.pl which explain this in more detail and include a small C program you can run to determine the correct settings for your machine.

The host name lookup (-r option) and relaying between hosts (-e and -l options) make extensive use of Unix networking and process control facilities. They are unlikely in the extreme to work on non-Unix systems, at least not without a great deal of modification.

The -r option, which attempts to translate numeric Internet addresses into host names, can cripple a heavily loaded system by a flood of name server requests. Read the discussion of whether this option is appropriate to use on your system in the OPTIONS section above before enabling it.

When logtail begins monitoring a new cycle of a file, entries written between the time the file was cycled and the time logtail transferred to monitoring it are not printed; only subsequently added entries will be seen. This is deliberate: the HTTP log of a busy Web site, for example, can grow to a very substantial size in the interval between being cycled and logtail's discovering it. Processing all the backed up items may be very time-consuming, and relaying them to other hosts may run into maximum network packet size restrictions which would require substantial additional complexity to work around. Since logtail is intended to provide an overview of system activity, a few overlooked messages on those infrequent occasions when files are cycled doesn't seem unreasonable.

FILES

SEE ALSO

Download logtail.tar.gz (gzipped TAR archive)

AUTHOR

This software is in the public domain. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, without any conditions or restrictions. This software is provided "as is" without express or implied warranty.