At work we have a few scripts that we would like to monitor, and define a few messages that should trigger an SMS to be sent to the person on call.

As most of these scripts log to syslog, and we have a our linux servers set up to forward syslog messages to a central log host, we going to monitor the central syslog for important messages originated by the scripts. In order to avoid message storms, however, we need a way of throttling duplicate messages. On our central log host we’re running swatch for real time analyzis of the incoming syslog messages.

In the below example, I’ve made a simple regular expression that extract kind of the “primary key” from the messages format I’m planning on having my scripts to use. So if we for example have a syslog message “2011-04-13T12:25:31.194694+02:00 myserver root: script myscript warning This is a warning message”, I would consider the “primary key” to be “myserver root: script myscript warning” and throttle on this value. The messages are throttled to avoid message storms, and they are piped into a separate files for script log messages.

Maybe I’ll extend the regular expression and log format to include a error code too later on.