Trying to debug rsyslog.conf issues? This can be hard. Learn here why, and how you can avoid problems. Did you know? Upon startup, rsyslog reads its config file, usually located in /etc/rsyslog.conf. While doing so, it can include config snippets usually located in /etc/rsyslog.d. But no matter if given directly inside the rsyslog.conf, or inside a snippet, the overall config is just a single text document made up of the main config file and the snippets. This is important to note when debugging rsyslog problems.
Config snippets – intended use and current abuse
First of all, distros are overdoing the use of snippets. Originally these were just meant to provide a way to other packages to add some very limited and important things to the rsyslog config (like an additional unix socket to listen to). Especially Ubuntu has created an art of “rsyslog config obfuscation” by using multiple snippets for just regular things. User have taken up on that, and create their own snippets as well. In the end result, this makes it very hard to understand what rsyslog sees and how rsyslog actually is executes the config.
How does rsyslog handle config snippets (from /etc/rsyslog.d)?
For rsyslog, only a single config file exists. If it finds an IncludeConfig statement, it merges all related config files into a single text “file”. Actually, it’s not a file, but an in-memory text document, but that’s a detail you can ignore.
From a practical point of view, you can understand what rsyslog does with this picture:
- It takes the main config file, and goes through it and all snippets. It merges all snippets, at the appropriate location, into a single text “file”. I call this file the “consolidated” config.
- Once rsyslog has created that single text file, it begins to process it. Note that at this point rsyslog does no longer know where the text originated from – from the main config or a snippet. It’s simply irrelevant to it.
- During processing, rsyslog interprets the config from top to bottom. Early things happen before later things. Again: which snippet the config was originally in, does not matter. It’s just the consolidated config that counts.
Problems with using to many rsyslog config snippets
The excessive use of snippets can have unexpected results. A typical problem is that in some snippet there is an unconditional stop statement. Processing stops at this point. Rsyslog does not any further process that message. That’s what you told rsyslog! It does not matter if any more statements are present in later snippets.
To aid users find such issues, rsyslog emits a warning messages when it finds an unconditional stop with additional config statements behind it. But it cannot detect a logic error with a selector line like this:
local3.* /var/log/myapp.log &
stop
If something else should happen with local3 messages, and that something is after this part of the config, it will not happen. Note that this is a typical problem I see when troubleshooting users issues. These kind of issues are very hard to debug when in separate snippets.
How to debug in such situations?
An essential tool is seeing the consolidated rsyslog.conf just like rsyslog sees it. Also, it is important to see any config errors rsyslog detect. Interestingly, many distributions simply throw away rsyslog messages, so a simple typo can become a nightmare.
Fortunately, help is easily at hand. Rsyslog has the ability to check the configuration and created the consolidated rsyslog.conf with a simple command line option. It is:
$ sudo rsyslogd -N1 -o /tmp/consolidated-rsyslog.conf -f /etc/rsyslog.conf
Execute this in a command line prompt (terminal). You can do this while rsyslog currently runs in the background. The command checks the configuration file.
The image above shows such a check run. Here, rsyslog detected an error, because it cannot find a module. It lets you know the approximate location inside the config file. Note the “on or before” part of the error message. This is literally meant, the actual issue may have occurred some lines above. Sometimes even in a different config snippet!
What we see in the screenshot is a typical typo. We can strongly guess that “imtcp” was meant and the “m” just got lost on the keyboard. Even if no error shows up in the config check, rsyslog creates the consolidated rsyslog.conf. You can now review it and learn the actual order of config statements – that is often surprising.
Note that the rsyslog community frequently requests a consolidated rsyslog.conf. With just a snippet, it is extremely hard to debug problems as we do not see the “real thing”.
Some general advise…
If there is no hard reason for using config snippets, stay away from them. It is always simpler and easier to understand if all config changes are made directly in rsyslog.conf, and at the place where they belong.
The only valid reason I see for snippet use is in large organizations, where different teams are responsible for different parts of the configuration. In that case, I highly recommend to make the snippets really self-contained, by properly using rulesets and other advanced concepts.
And, of course, snippets are totally fine for what they were intended: adding small bits of functionality for a different package during package installation.
Happy (r)syslogging!