Category: rsyslog.conf

Trying to debug rsyslog.conf issues? This can be hard. Learn here why, and how you can avoid problems. Did you know? Upon startup, rsyslog reads its config file, usually located in /etc/rsyslog.conf. While doing so, it can include config snippets usually located in /etc/rsyslog.d. But no matter if given directly inside the rsyslog.conf, or inside a snippet, the overall config is just a single text document made up of the main config file and the snippets. This is important to note when debugging rsyslog problems.

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I’m mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I’m new, I’d considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner – I learn best by example.
Are there any serious downsides to NOT going with the most current syntax?

The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn’t carry over for good reasons – this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That’s actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people’s knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

In 7.3.2, to be released today, I  reworked ratelimiting and “last message repeated n times” handling –  all over rsyslog code. Each of the supported inputs now supports  linux-like ratelimiting (formerly only imuxsock did). Also, the “last message repeated n times” is now processed at the input side  and no longer at the output side of rsyslog processing. This  provides the basis for new future additons as well as usually more performance and a much simpler output part, which can be even further refactored. I kept away from refactoring that part of the code as I think the change was already large enough and I’d like to get some practical feedback on that version first.

Of course, the input side now has become a bit more complex, but this offers great new abilities. Most importantly, the “last message repeated n times” is finally a bit useful at all, because the repeat message detection now happens on a per-input basis. Previously, messages from different inputs were mixed and as such repeat messages from a single input could not reliably be detected and prevented. That was a design limitation stemming back from the original handling. It finally was time to remove that limitation.

Please note that we now also support linux-type ratelimiting, usually via the two parameters “ratelimit.burst” and “ratelimit.interval”. The “ratelimit.burst” specifies how many messages can be emitted at most in the period specified by “ratelimit.interval” (in seconds). Any excess messages are discarded (the idea of just delaying them is floating around, but this is not yet implemented – but fairly easy to do…). At the end of the interval, the whole processing is restarted. This mode is MUCH faster than “last message repeated n times” but of course does not just reduce similiar messages.

A concrete example for imudp looks like this:

input(type="imudp" port="514"
      ratelimit.burst="5000" ratelimit.interval="2")

This allows at most 5,000 messages within 2 seconds.

Rsyslog 7.2+ introduced a couple of cool config enhancements, among them a new way to specify rulesets and to call into a ruleset (a much better replacement for omruleset). Unfortunatley, the doc is currently extremely sparse. That will change soon, but in the mean time I thought I provide at least some clues here via the blog.

In essence, the ruleset statement permits to specify a ruleset. It’s written as

ruleset(name="rulesetname") { statements here }

Rulesets can be bound to inputs, as usual, and any ruleset can call into another ruleset via the call statement:

call rulesetname

Note that in call, the rulesetname is just plainly specified. We created it that way as we thought this looks most intuitively to people used to any kind of scripting or programming language.

A somewhat larger sample (bascially taken from the rsyslog mailing list, thanks to Brian Knox) is:

module(load="imptcp" keepalive="on")
# use imptcp just as example for bind ruleset below
ruleset(name="rs1") {
        *.* /var/log/test1.log
}
ruleset(name="rs2") {
        *.* /var/log/test2.log
        call rs1
}
input(type="imptcp" port="13514" ruleset="rs2")

All statements NOT specified inside a ruleset become part of the default ruleset. Legacy-style $Ruleset statements are still supported, but cannot be used inside a ruleset() statement (that would create a big mess and is totally unnecessary).

In order to aid some string comparisons, I have just introduced constant string arrays in rsyslog. Look at this construct:

if $msg contains [“event12”, “event15″,”event19”] then
    /home/rgerhards/logfile

This will match if $msg contains any of the three strings given inside the array, so it works like an implicit “or”. I have seen this in a couple of customer configs and I am confident that this will ease rsyslog configuration. In later releases, it will also provide room for performance optimizations.

This feature will be part of 7.1.7.

[edit] Starting with rsyslog 7.3.6, string array comparisons are notably faster, especially if the array is large. At most, there are log(n) comparisons required (where n is the array size).

I have just completed the new input() config statement. This concludes the major part of the new config format for v6 (v7 will also support an enhanced ruleset() statement). I wanted to give you some quick ideas of how the new format looks in practice. So I have created a small test rsyslog.conf with the old-style directives commented out and followed by the new style ones. I guess it is pretty self-explanatory. Here it is:

#$ModLoad imfile
#$inputfilepollinterval 1
module(load=”imfile” pollingInterval=”1″)

#input(type=”imuxsock” )
module(load=”imuxsock” syssock.use=”off”)
input(type=”imuxsock” socket=”/home/rgerhards/testsock”)

$ModLoad imfile
module(load=”imfile”)
#$InputFileName /tmp/inputfile
#$InputFileTag tag1:
#$InputFileStateFile inputfile-state
#$InputRunFileMonitor
input(type=”imfile” file=”/tmp/inputfile” tag=”tag1:” statefile=”inputfile-state”)

module(load=”imptcp”)
#$InputPTCPServerRun 13514
input(type=”imptcp” port=”13514″)

module(load=”imtcp” keepalive=”on”)
#$InputTCPServerSupportOctetCountedFraming off
#$InputTCPServerInputName tcpname
#$InputTCPServerRun 13515
input(type=”imtcp” port=”13515″ name=”tcpname” supportOctetCountedFraming=”off”)

#$UDPServerRun 13514
#$UDPServerRun 13515
input(type=”imudp” port=”13514″)
input(type=”imudp” port=”13515″)

Rsyslog supports BSD-style blocks since ages. They were a pretty handy tool to group actions together that should act only on remote hosts or log messages from specific programs. However, the v7 config system with its full nesting capabilities provides a much better – and easy to use – way to specify this. If both systems are mixed, the problem is that BSD-style blocks can be used to violate the nesting structure (examples below). Also, support for them adds a lot to rule engine code complexity. And finally, they are also very seldom used, few users even know they exist.

As a result, I have decided to drop support for BSD-style blocks in rsyslog v7 and above. A poll on the mailing list a week ago did not make anybody speak up against that change. So I assume none is hurt. This is especially the case as the conversion of BSD-style blocks to nested config is a very easy one.

Let’s look at this example:

mail.* :mmjsonparse:

In pre-v6, action queues are quite hard to understand, because the syntax is very counter-intuitive. Based on a recent conversation, I thought I share a little bit of advise on how queues work.

Each action has a dedicated queue. This queue can be in-memory, it can be on disk and it can be a combination of both. There is also direct mode, which means by design there is a queue but the actual driver does forward messages to the action without action queueing.

When you define actions, the queue parameters belong to the next action (in v6 this is far easier to understand and see as the queue params are specified within the action). In the snippet below, we have two actions, and each of them is configured with different queue settings. This results with two different sets of queue files being written. It is important that the queue file names are different (as they are), but otherwise the queues are independent. There can be as many disk (or DA) queues as there are actions. Note that if there is only a single action queue and n actions, this means only one action uses the disk queue, whereas n-1 actions do not use it (if nothing else is set, they run in direct mode).

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName nfsq    # set file name, also enables disk mode
*.* /var/log/log1

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName diskq    # set file name, also enables disk mode
*.* /var/log/log2

*.* /var/log/log3 # DIRECT mode here!!!!

In v6, queue parameters are set directly within the action:

*.* action(type="omfile" 
       queue.filename="nfsq" queue.type="linkedlist"
       file="/var/log/log1")

*.* action(type="omfile" 
       queue.filename="diskq" queue.type="linkedlist"
       file="/var/log/log2")

*.* action(type="omfile" file="/var/log/log3")
 

The v6 format hopefully make more clear what belongs together.