rsyslog: how to debug rsyslog.conf problems?

Trying to debug rsyslog.conf issues? This can be hard. Learn here why, and how you can avoid problems. Did you know? Upon startup, rsyslog reads its config file, usually located in /etc/rsyslog.conf. While doing so, it can include config snippets usually located in /etc/rsyslog.d. But no matter if given directly inside the rsyslog.conf, or inside a snippet, the overall config is just a single text document made up of the main config file and the snippets. This is important to note when debugging rsyslog problems.

Some Distributions (here: Ubuntu) and users split the rsyslog.conf file in too many snippets. This often causes hard-to-debug problems. (Screenshot: Rainer Gerhards)
Continue reading “rsyslog: how to debug rsyslog.conf problems?”

Should I use rsyslog’s new or old config style?

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I’m mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I’m new, I’d considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner – I learn best by example.
Are there any serious downsides to NOT going with the most current syntax?

The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn’t carry over for good reasons – this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That’s actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people’s knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

multi-character field delimiters

On the rsyslog mailing list, the ability to use multiple characters as field delimiters had been requested recently. Today, I took some time off the my schedule and implemented that functionality. It is probably very useful for a number of cases. An important one is probably in combination with control character escaping, where rsyslog by default expands a single character into a four-byte escape “#ooo” with o being the octal character code (so  e.g. US ASCII HT [horizontal tab] becomes “#011”).

The new functionality is available for the RainerScript field() function. I do not intend to add it to template strings.

Some quick usage sample:

The following is the traditional way of single-byte delimiters, here with the comma character (US ASCII decimal code 44):

set $!usr!field2 = field($msg, 44, 2);
template (name=”fld” type=”string” string=”‘%$!usr!field2%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

And this is the same with the string “#011” as delimiter:

set $!usr!field2 = field($msg, “#011”, 2);
template (name=”fld” type=”string” string=”‘%$!usr!field2%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

Note that the field number (index) need not necessarily to be fixed. It can be derived from an appropriately formatted message. Here the first field contains the actual field to extract, delimiter is “#011” again:

set $!usr!idx = field($msg, “#011”, 1);
set $!usr!field = field($msg, “#011”, $!usr!idx);
template (name=”fld” type=”string” string=”‘%$!usr!field%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

In that last sample the $msg of

“3#011val 1#011val 2#011val 32#val 4”

would return

“val 2”

Keep in mind that the first field is the field index, so the actual data fields start at 2 (field 1 is “3”, field 2 is “val 1”, field 3 “val 2” and so on).

This functionality is already present in git master head and will be released as part of 7.3.7 in the not so distant future. Some more details can be found inside the RainerScript documentation page.

rate-limiting in rsyslog 7.3.2

In 7.3.2, to be released today, I  reworked ratelimiting and “last message repeated n times” handling –  all over rsyslog code. Each of the supported inputs now supports  linux-like ratelimiting (formerly only imuxsock did). Also, the “last message repeated n times” is now processed at the input side  and no longer at the output side of rsyslog processing. This  provides the basis for new future additons as well as usually more performance and a much simpler output part, which can be even further refactored. I kept away from refactoring that part of the code as I think the change was already large enough and I’d like to get some practical feedback on that version first.

Of course, the input side now has become a bit more complex, but this offers great new abilities. Most importantly, the “last message repeated n times” is finally a bit useful at all, because the repeat message detection now happens on a per-input basis. Previously, messages from different inputs were mixed and as such repeat messages from a single input could not reliably be detected and prevented. That was a design limitation stemming back from the original handling. It finally was time to remove that limitation.

Please note that we now also support linux-type ratelimiting, usually via the two parameters “ratelimit.burst” and “ratelimit.interval”. The “ratelimit.burst” specifies how many messages can be emitted at most in the period specified by “ratelimit.interval” (in seconds). Any excess messages are discarded (the idea of just delaying them is floating around, but this is not yet implemented – but fairly easy to do…). At the end of the interval, the whole processing is restarted. This mode is MUCH faster than “last message repeated n times” but of course does not just reduce similiar messages.

A concrete example for imudp looks like this:

input(type="imudp" port="514"
      ratelimit.burst="5000" ratelimit.interval="2")

This allows at most 5,000 messages within 2 seconds.

how to use rsyslog’s ruleset and call statements

Rsyslog 7.2+ introduced a couple of cool config enhancements, among them a new way to specify rulesets and to call into a ruleset (a much better replacement for omruleset). Unfortunatley, the doc is currently extremely sparse. That will change soon, but in the mean time I thought I provide at least some clues here via the blog.

In essence, the ruleset statement permits to specify a ruleset. It’s written as

ruleset(name="rulesetname") { statements here }

Rulesets can be bound to inputs, as usual, and any ruleset can call into another ruleset via the call statement:

call rulesetname

Note that in call, the rulesetname is just plainly specified. We created it that way as we thought this looks most intuitively to people used to any kind of scripting or programming language.

A somewhat larger sample (bascially taken from the rsyslog mailing list, thanks to Brian Knox) is:

module(load="imptcp" keepalive="on")
# use imptcp just as example for bind ruleset below
ruleset(name="rs1") {

        *.* /var/log/test1.log
ruleset(name="rs2") {
        *.* /var/log/test2.log
        call rs1
input(type="imptcp" port="13514" ruleset="rs2")

All statements NOT specified inside a ruleset become part of the default ruleset. Legacy-style $Ruleset statements are still supported, but cannot be used inside a ruleset() statement (that would create a big mess and is totally unnecessary).

rsyslog string array comparisons

In order to aid some string comparisons, I have just introduced constant string arrays in rsyslog. Look at this construct:

if $msg contains [“event12”, “event15″,”event19”] then

This will match if $msg contains any of the three strings given inside the array, so it works like an implicit “or”. I have seen this in a couple of customer configs and I am confident that this will ease rsyslog configuration. In later releases, it will also provide room for performance optimizations.

This feature will be part of 7.1.7.

[edit] Starting with rsyslog 7.3.6, string array comparisons are notably faster, especially if the array is large. At most, there are log(n) comparisons required (where n is the array size).

rsyslog’s new input() statement: a quick look

I have just completed the new input() config statement. This concludes the major part of the new config format for v6 (v7 will also support an enhanced ruleset() statement). I wanted to give you some quick ideas of how the new format looks in practice. So I have created a small test rsyslog.conf with the old-style directives commented out and followed by the new style ones. I guess it is pretty self-explanatory. Here it is:

#$ModLoad imfile
#$inputfilepollinterval 1
module(load=”imfile” pollingInterval=”1″)

#input(type=”imuxsock” )
module(load=”imuxsock” syssock.use=”off”)
input(type=”imuxsock” socket=”/home/rgerhards/testsock”)

$ModLoad imfile
#$InputFileName /tmp/inputfile
#$InputFileTag tag1:
#$InputFileStateFile inputfile-state
input(type=”imfile” file=”/tmp/inputfile” tag=”tag1:” statefile=”inputfile-state”)

#$InputPTCPServerRun 13514
input(type=”imptcp” port=”13514″)

module(load=”imtcp” keepalive=”on”)
#$InputTCPServerSupportOctetCountedFraming off
#$InputTCPServerInputName tcpname
#$InputTCPServerRun 13515
input(type=”imtcp” port=”13515″ name=”tcpname” supportOctetCountedFraming=”off”)

#$UDPServerRun 13514
#$UDPServerRun 13515
input(type=”imudp” port=”13514″)
input(type=”imudp” port=”13515″)

setting variables in rsyslog v7

Starting with rsyslog 7.1.3, variables can be manipulated inside rsyslog.conf. Note that the native message properties (like $msg) cannot be modified, but the CEE/lumberjack-type properties (those $!name variables) can. As variable modification is primarily an aid for template generation and modification, we do not consider this a restriction, as native message properties can simply be copied to the CEE tree if this is needed.

Note that CEE/lumberjack properties, as implemented in rsyslog,  can be hierarchical and levels are delimited by the bang sign (based on lumberjack recommendations). So “!uid” is the uid field in the CEE root, whereas “!usr!uid” is the uid field inside the usr container. Nesting can be as deep as desired. Currently, all parts of the CEE tree can be accessed. In later versions, this may require the setting of a global option. If that will happen depends on the feedback we receive.

So: how does it work?

To set a variable, simply use
set varname = expression;
Where expression can be an arbitrary complex expression, just like in an “if” statement. Note the semicolon at the end: this is unfortunately necessary and a diversion from the other config statements. However, this is the price we need to pay to remain upward compatible with the old style config format.

Concrete examples:

set $!usr!level2!var1 = “test”; 
set $!usr!level2!var1 = $msg; # update variable with native MSG field
set $!usr!level2!var2 = 1+1; # set !usr!level2!var2 = 2
set $!usr!level2 = $fromhost; # invalid

The last example is invalid and will not be executed because it tries to replace a complete container (!usr!level2) with a single value. This is rejected and not executed. Note that this problem is not detected during config read time but rather during execution time (in less trivial samples, it can not be reliable detected without execution).

Note that string concatenation is currently NOT supported, but it will be soon in the next releases. Also, full JSON containers cannot yet be copied. If this is tried to, the resulting variable will receive a string representation, which most probably is not what you wanted (and you will get a different result in future versions).

There is also an accompanying “unset” statement to remove a variable that is no longer needed. This is primarily meant to restructure a CEE container. It’s syntax simply is:

unset varname;

Again, note the semicolon at the end. A concrete example is

unset !usr!level2!var1;

which removes a single element. But full containers can also be removed:

unset !usr!level2;

Note that all variables are assigned to the message currently being processed. There currently is no way to set global variables (but this may become available based on feedback I see).

BSD-Style blocks will go away in rsyslog v7

Rsyslog supports BSD-style blocks since ages. They were a pretty handy tool to group actions together that should act only on remote hosts or log messages from specific programs. However, the v7 config system with its full nesting capabilities provides a much better – and easy to use – way to specify this. If both systems are mixed, the problem is that BSD-style blocks can be used to violate the nesting structure (examples below). Also, support for them adds a lot to rule engine code complexity. And finally, they are also very seldom used, few users even know they exist.

As a result, I have decided to drop support for BSD-style blocks in rsyslog v7 and above. A poll on the mailing list a week ago did not make anybody speak up against that change. So I assume none is hurt. This is especially the case as the conversion of BSD-style blocks to nested config is a very easy one.

Let’s look at this example:

*.* /var/log/prog1.log
*.* /var/log/prog1again.log
*.* /var/log/prog2.log
*.* /var/log/prog2again.log
This code can very simply be replaced by:
 if $programname == ‘prog1’ then {
if $programname == ‘prog2’ then {
And if you prefer the more powerful action statments (probably not so much benefit for this use case…), you can write:
 if $programname == ‘prog1’ then {
   action(type=”omfile” file=”/var/log/prog1.log”)
   action(type=”omfile” file=”/var/log/prog1again.log”)
if $programname == ‘prog2’ then {
   action(type=”omfile” file=”/var/log/prog2.log”)
   action(type=”omfile” file=”/var/log/prog2again.log”)
I expect that usually these easy cases happen. HOWEVER, if I had kept support for BSD-style blocks, one could configure things like this:
if $msg contains ‘test’ then  {
                action(type=”omfile” file=”/var/log/somefile”)

mail.* :mmjsonparse:

                               & action(type=”omfile” file=”/var/log/somefile2″)


                               & ~
                if $msg contains ‘test2’ then
Can you make out the actual nesting structure of this config? When, for example, programname needs to be “prog3” and what happens then? IMHO this combination can cause considerable user confusion and frustration. As such, I made a sharp cut and removed it.
My apologies for those that need to do the manual conversion. I am sure the time is well-invested in the long term.

rsyslog action queue

In pre-v6, action queues are quite hard to understand, because the syntax is very counter-intuitive. Based on a recent conversation, I thought I share a little bit of advise on how queues work.

Each action has a dedicated queue. This queue can be in-memory, it can be on disk and it can be a combination of both. There is also direct mode, which means by design there is a queue but the actual driver does forward messages to the action without action queueing.

When you define actions, the queue parameters belong to the next action (in v6 this is far easier to understand and see as the queue params are specified within the action). In the snippet below, we have two actions, and each of them is configured with different queue settings. This results with two different sets of queue files being written. It is important that the queue file names are different (as they are), but otherwise the queues are independent. There can be as many disk (or DA) queues as there are actions. Note that if there is only a single action queue and n actions, this means only one action uses the disk queue, whereas n-1 actions do not use it (if nothing else is set, they run in direct mode).

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName nfsq    # set file name, also enables disk mode
*.* /var/log/log1

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName diskq    # set file name, also enables disk mode
*.* /var/log/log2

*.* /var/log/log3 # DIRECT mode here!!!!

In v6, queue parameters are set directly within the action:

*.* action(type="omfile" 
queue.filename="nfsq" queue.type="linkedlist"
*.* action(type="omfile" 
queue.filename="diskq" queue.type="linkedlist"

*.* action(type="omfile" file="/var/log/log3")

The v6 format hopefully make more clear what belongs together.