Using swatch to throttle script log messages
At work we have a few scripts that we would like to monitor, and define a few messages that should trigger an SMS to be sent to the person on call.
As most of these scripts log to syslog, and we have a our linux servers set up to forward syslog messages to a central log host, we going to monitor the central syslog for important messages originated by the scripts. In order to avoid message storms, however, we need a way of throttling duplicate messages. On our central log host we’re running swatch for real time analyzis of the incoming syslog messages.
In the below example, I’ve made a simple regular expression that extract kind of the “primary key” from the messages format I’m planning on having my scripts to use. So if we for example have a syslog message “2011-04-13T12:25:31.194694+02:00 myserver root: script myscript warning This is a warning message”, I would consider the “primary key” to be “myserver root: script myscript warning” and throttle on this value. The messages are throttled to avoid message storms, and they are piped into a separate files for script log messages.
1 2 3 4 5 |
perlcode my $script_regexp = '(\s\w+\s\w+:\sscript\s\w+\s(critical|warning))'; watchfor /$script_regexp/ pipe "cat - >> /var/log/scripts-output.log; echo '' >> /var/log/scripts-output.log" threshold track_by=$2, type=limit, count=1, seconds=30 |
Maybe I’ll extend the regular expression and log format to include a error code too later on.