rsyslog disk queues: refactor or redesign? – part I

I am currently thinking about the design of rsyslog’s disk queues. There are some things that I really like about them, but there are also lots of things I dislike. This blog post is meant to save as a basis for discussing whether the disk queue system should better be refactored or completely redesigned. The actual discussion should please be kept on the mailing list. I just blogged because this makes it easier to have a stable reference on where that whole thing started. This is the first in a series of ? blog posts. It describes the current state as of now (rsyslog 7.3.1 and below).

First of all, it must be noted that both serious refactoring as well as redesign are rathter costly. So I don’t know when I can actually do that – it probably depends on the fact if there are some sponsors at least for parts of the work available. With that said, let’s get down to technology:

The disk queue system is designed to be used primarily in exceptions (like remote destination being down for a longer period of time) or for high-reliability environments. A main design goal was that they should not take too much disk space, at least not more than necessary at any point in time. Also, human operators should be able to fix things (or at least understand them), when things go really wild.

To solve these needs, queue data is stored in pure US-ASCII text form, without any kind of compression or binary storage. The encoding used is rather simple it’s basically name, type and value of any property, one per line (the details can be found in the relevant source files). The “not too much space” requirement is solved by using a set of rotating sequential files, where each file has a configured maximum size (10MB by default) [to be technically precise, the maximum is not totally enforced, a complete message is persisted to the file, even if that goes slightly above the maximum]. There is a kind of head and tail pointer into that rotating file set: the head is used by new messages coming in. They are always appended to the most recent file. The tail pointer is where messages are dequeued from, it alwass points to the oldest file inside the file set. Originally, there were just these two pointers, and a file was deleted as soon as the tail pointer moved past the end of the current file. However, this was modified to cover the ultra-reliable use case in v5: The problem with two pointers was that when messages were dequeued at the end of the tail file, that file was immediately deleted. When now something went wrong, the potentially unprocessed messages were lost. To prevent this, v5+ has actually three pointers: the head pointer (as before) a read pointer (former tail) and a delete pointer (kind of “real tail”). The read pointer is now used to dequeue the messages, and the delete pointer is used to re-read messages from the file after they have been processed and delete queue files, if they go out of the current file set while doing so (this actually is exactly the processin that the former tail pointer did). The bottom line is that each message is written once but read twice during queue processing.  The queue also maintains a state (or “queue info” – .qi) file, which among others contains these pointers. The user can configure how often data shall be synced to disk. The more often, the more robust the queue is, but it also gets slower as part of that process. To flush, both the current write needs to be flushed as well as the state file. For the state file, this currently means it is opened, written and closed again. At the extreme, this happens for each record being written (but keep in mind that the queue file itself is always kept open). It is also worth mentioning that all records are of variable size, so there is no random access possible to messages that reside inside the queue file.

From the threading point of view, queue operations are naturally serial (as sequential files are involved). As such, a queue usues exactly one worker, no matter how many worker threads the user has configured. In case of a disk assisted (DA) queue, there are as many workers as configured for the in-memory part of the queue and one worker for the disk part of the queue. Note that all of these workers run in parallel. That also means that message sequence will change if disk and in-memory mode run at the same time But this comes at no surprise: for obvious reasons, this is always the case when multiple workers process queue messages in parallel (otherwise, the parallel workers would need to serialize the, effectively running a single worker thread…). To keep the code simple later versions of rsyslg (late v5+) keep the disk queue open once it has been started. First of all, this removes some complex synchronization issues inside the rsyslog core, which would otherwise not only complicate the code but slow things down, even when no disk queue is active. Secondly, there is a good chance that disk mode is needed again if it was initiated once, so it probably is even very smart to make this as painless as possible. Note that during rsyslog shutdown, the disk queue files are of course fully be deleted.

rate-limiting in rsyslog 7.3.2

In 7.3.2, to be released today, I  reworked ratelimiting and “last message repeated n times” handling –  all over rsyslog code. Each of the supported inputs now supports  linux-like ratelimiting (formerly only imuxsock did). Also, the “last message repeated n times” is now processed at the input side  and no longer at the output side of rsyslog processing. This  provides the basis for new future additons as well as usually more performance and a much simpler output part, which can be even further refactored. I kept away from refactoring that part of the code as I think the change was already large enough and I’d like to get some practical feedback on that version first.

Of course, the input side now has become a bit more complex, but this offers great new abilities. Most importantly, the “last message repeated n times” is finally a bit useful at all, because the repeat message detection now happens on a per-input basis. Previously, messages from different inputs were mixed and as such repeat messages from a single input could not reliably be detected and prevented. That was a design limitation stemming back from the original handling. It finally was time to remove that limitation.

Please note that we now also support linux-type ratelimiting, usually via the two parameters “ratelimit.burst” and “ratelimit.interval”. The “ratelimit.burst” specifies how many messages can be emitted at most in the period specified by “ratelimit.interval” (in seconds). Any excess messages are discarded (the idea of just delaying them is floating around, but this is not yet implemented – but fairly easy to do…). At the end of the interval, the whole processing is restarted. This mode is MUCH faster than “last message repeated n times” but of course does not just reduce similiar messages.

A concrete example for imudp looks like this:

input(type="imudp" port="514"
      ratelimit.burst="5000" ratelimit.interval="2")

This allows at most 5,000 messages within 2 seconds.

how to use rsyslog’s ruleset and call statements

Rsyslog 7.2+ introduced a couple of cool config enhancements, among them a new way to specify rulesets and to call into a ruleset (a much better replacement for omruleset). Unfortunatley, the doc is currently extremely sparse. That will change soon, but in the mean time I thought I provide at least some clues here via the blog.

In essence, the ruleset statement permits to specify a ruleset. It’s written as

ruleset(name="rulesetname") { statements here }

Rulesets can be bound to inputs, as usual, and any ruleset can call into another ruleset via the call statement:

call rulesetname

Note that in call, the rulesetname is just plainly specified. We created it that way as we thought this looks most intuitively to people used to any kind of scripting or programming language.

A somewhat larger sample (bascially taken from the rsyslog mailing list, thanks to Brian Knox) is:

module(load="imptcp" keepalive="on")
# use imptcp just as example for bind ruleset below
ruleset(name="rs1") {

        *.* /var/log/test1.log
ruleset(name="rs2") {
        *.* /var/log/test2.log
        call rs1
input(type="imptcp" port="13514" ruleset="rs2")

All statements NOT specified inside a ruleset become part of the default ruleset. Legacy-style $Ruleset statements are still supported, but cannot be used inside a ruleset() statement (that would create a big mess and is totally unnecessary).

new v7/v6 stable, v5 now legacy

Today is a very important day for me – and rsyslog. After a very intense work period, the first v7 version is finally ready for the “stable” label. It contains a lot of new and important functionality. IMHO v7 is the most important release since the introduction of v4, and the next 7.3+ versions will bring even more very important enhancements. We ironed out quite some bugs during the beta phase. Interestingly, some of the bug fixes stem back to as far as v5.10, which recently received quite some attention.

As v7-beta is based on v6-beta, we can also release a new v6-stable, because v6 is a proper subset and all relevant patches have been applied to v6 as well. So they receive kind of “transitive readiness” via v7. It is important to note that we developed concepts in v6, but usually published only the merged-up version in v7 (in order to save development effort). As such, the release count for v6 (6.5.1) doesn’t look like it is a version ready for the “stable” label, but in fact the release count is artificially low due to the way we worked on the v6/v7 tandem.

Let me stress again that you usually want to go for v7 to see the latest and greatest in logging. Version 6 is basically a v5 engine, somewhat performance enhanced and with experimental support for structured logging/lumberjack logging. If you want to use structured logging in production, v7 is highly recommended. For example v6 does NOT support disk-queues in combination with structured logging, whereas v7 of course does.

With all these good news, v5 is now legacy and will no longer be officially supported by the project. Full support of course is still available as part of Adiscon’s rsyslog support packages. In regard to its wide use in distributions, we will still collaborate with the distro maintainers to solve problematic bugs (as we always did in the past for officially non-supported versions). However, this still means we need to care much less about v5, and that in turn means we can free many development ressources tied to maintaining old stuff and merging all those changes to newer versions. This has been very time-consuming in the past and I am very relieved that we currently have a plain v6 stable, v7 stable and an active development branch taking place in v7, only. I hope this will remain the case (except for some occasional v7-betas) at least for the next couple of month. It would definitely boost my productivity.

Todays releases are the culmination of a group effort. I would like to express special thanks to Miloslav Trmač, Milan Bartos, David Lang and Michael Biebl for their good suggestions, motivation and code contributions. I also thank the many other contributors (just have a look at the ChangeLog or git history!). The project would have been unable to reach this important milestone without all your contributions, including the well-done bug reports! Thanks a lot, folks!

With that said, I hope everyone enjoys the new stable versions. Also, look forward to an exciting future, which already has begun with the 7.3 series, which contains some good performance improvements. We continue to work on many cool new features!

new “stable” policy for rsyslog?

For many years, I have strongly opted for the devel-beta-release cycle for rsyslog, where a transition from beta to release only happens after two to three month in order to be sure that a stable version really was stable. This worked well, because early adopters of the beta branches helped to iron out any remaining bugs. Unfortunately, I see this the early adoption rate decline for around two years now (actually starting with v5). The reason is probably obvious: rsyslog has become so feature-rich that users usually do not see any need to work with the beta versions but rather wait for stable. The bottom line is that when everybody waits for stable, the beta won’t stabilize at all ;-).

Unfortunately, I have seen this, too, happen in practice: even though we had very lengthy beta periods, problem reports just came in after the beta has been declared to be stable. At that time, many folks seem to start using the system, and as nobody did any testing before, things then begin to break. I guess there is noone to blame, that’s just how things work. However, one needs to know that I invest quite some time into the devel-beta-stable cycle. Time, that is most probably better spent at other things, if looked at in the light of what I just described.

As such, I will most probably change our “stable” policy for rsyslog soon: the tag “stable” will no longer mean that the product has actually stabilized, but that we are ready to support it as a stable version (probably very important to those folks with support contracts). While this may sound harsh, it has a lot of good: first of all, this is what actually happens for 2+ years already, just with a 3-month delay. So we spare these months, what means new features will become more readily available, and bugs are probably quicker found AND fixed (because if a bug reports is closer to implementation, it is usually easier to figure out what is wrong). With that said, rsyslog 7.2 will probably the last stable that went through the usual cycle.

If you have a great idea how we can actually get more beta deployments, I’ll reconsider that decision.

Note that we will still have devel-beta-stable branches, but beta will much quicker turn into stable than before.

Main Advantages of rsyslog v7 vs. v5

A lot of work has been done since the days of rsyslog v5. In this  post, I will provide the top 5 advantages of the v7 engine over the previous version. Please note that I do not talk about v6, as it is very close to the v5 engine, just with the improved config language. V6 also provides some experimental support for structured logging (project lumberjack), which has been fully matured in v7 (I strongly recommend to use v7 if you are serious about structured logging).

So, here are the main points:

  • greatly improved configuration language – the new language is much more intuitive than the legacy format. It will also prevent some typical mistakes simply be not permitting these invalid constructs. Note that legacy format is still fully supported (and you can of course do the same mistakes as before if you use legacy format).
  • greatly improved execution engine – with nested if/then/else constructs as well as the capability to modify variables during processing.
  • full support for structured logging and project lumberjack / CEE – this includes everything from being able to create, interpret and handle JSON-based structured log messages, including the ability to normalize legacy text log messages.
  • more plugins – like support for MongoDB, HDFS, and ElasticSearch as well as for the kernel’s new structured logging system.
  • higher performance – many optimizations all over the code, like 5 to 10 times faster execution time for script-based filters, enhanced multithreaded TCP input plugin, DNS cache and many more.

Of course, there are many more improvements. This list contains just the most important ones. For full details, check the file ChangeLog.