Author: Rainer Gerhards

There have been a couple of questions when rsyslog’s next stable release (7.6) will be released.

Originally, the idea was to have this done by the end of this year, which essentially means end of November. Unfortunately, this needs to be pushed into early January 2014. The reason is code quality.

In October, we discovered a larger regression and inconsistency in regard to the new variable support in rsyslog. As it turned out, the only good choice was to seriously refactor the code handling variables and message properties. Unfortunately, such a larger refactoring also means bug potential, so I don’t feel well to release 7.6 stable with just two weeks of testing on this refactored code. Looking forward, a release around mid-December sounds doable, but this means we’ll hit the holiday break with a brand-new stable release. This is something that I definitely want to avoid. As such, the current plan is to do the release shortly after the holiday break.

In order to keep this release window, I will also avoid larger modifications to the code base. That doesn’t mean things like better support for impstats support in omelasticsearch, but it may mean that the updated imfile code will probably not be merged into v7, at least not now.

Depending on how things work out with rsyslog v8, I will do another 7.7 devel series for less intrusive enhancements or move directly to v8 only. This will be also be decided around January 2014, when we hopefully got enough feedback on v8.

Do you remember last October? The rsyslog mailing list got very busy with things like output load balancing, global variables, and a lot of technical details of the rsyslog engine. I don’t intend to reproduce all of this here. The interested reader may simply review the rsyslog mailing list archives, starting at October 2013.

Most of these discussions focused on the evolution of the rsyslog engine. Essentially, the v7 engine brought many new things, but was constraint due to a lot of legacy, for example the output module interface which originally (roughly 8 years ago) was designed to keep it very simple to use, especially for developers who did not know about threading (which was uncommon to know at that time). Also, the engine gained lot’s of speeds by a special SIMD-like execution mode (details in mailing list archive), but that mode was counter-productive for some of the newer features. An ultimate problem were global variables, which were very hard to implement with the v7 engine. I even proposed a complete shadow variable system to make them fit into the framework, but this was -thankfully- not well received by the community at large.

Note that I wanted to re-write at least part of the core engine for quite some while, but it sounded like a task so large that I deferred it one time after another. However, seeing all the time going into the discussion, potential work-arounds (like shadow variables) and the then still-nonoptimal state I very strongly considered if it may be time for a radical change. Better spend the time on cool new things than investing that time into some “work-around” type of implementations.

With that on my mind, I had a couple of days of drafting and some talks with my peers. It turned out that we wanted to at least give this a try and see how (and if;)) things would evolve. So I did a week of test implementation, mostly focused on a new, highly concurrent output module interface. That did work out rather well, so we finally decided to go for the full length. I started rewriting the complete core rule execution engine.

And this brings us to the lucky number: let’s welcome rsyslog v8, as the new version will be named. I thought for a short moment to stick with v7, but the changes are really, really big and quite a bit of legacy and upward compatibility has been lost. For example, all output modules need to be rewritten. So while v8 may not be a great marketing tool to get it quickly into the distros, it correctly flags that some bigger changes are to be anticipated (but as usual we were very conservative with breaking existing configurations ;)).

I did not announce this effort any sooner as until now I wasn’t sure if it would really work out. Obviously, I now think thinks look sufficiently well ;-). It’s not yet deployable, but we are getting closer.

Expect a 8.1.0 release within the next couple of days. We are working on the final finishing touches, many of which involve testing. Note that 8.1.0 will be highly experimental and really is for folks who would like to get an impression of its new capabilities … and those trying to help find bugs.

There are some restrictions on supported modules and features, and some upgrades may run into trouble with that (details will be available together with the release). Also note that the rewrite is not yet 100% complete. Some of the finer details still need to be advanced, e.g. a new, higher performance and easier to use interface for output transactions. Or some aspects of message error processing … and so on. This follows our usual pattern that in a new development branch we bring in new features gradually, until we have reached the final goal. In any case, we are now pretty confident that we will reach this goal. If all goes exceptionally well, a v8-stable may come up in March 2014 (and free support for v7 will be available for another month or two so that there is time to migrate).

To facilitate this move, git branches will change. Yesterday’s master branch will become available as v7-devel (now in place) and the master branch will hold the new v8 code. Follow this blog post and the rsyslog mailing list to be sure to see exciting new versions when they become available.

Version 7-devel will get kind of a beta status, where mostly bug fixes are applied. However, some non-bugfix changes will also happen, but only after careful consideration. The focus obviously is now on v8 and this is where we will do the “cool” stuff in the future.

Note that the average user will possibly not notice much difference between v7 and v8 initially. The new engine will be most useful for power users and installations with complex rules and/or a high log volume.

I am very excited about this new branch of development and hope you’ll also be excited as it unrolls. Also my sincere thanks to everyone who waits patiently for v7 related work. I was too busy to look a that the past few weeks and will probably be occupied with v8 for a quite a bit longer.

From time to time, someone asks why rsyslog disk-assisted queues keep one file open until shutdown. So it probably is time to elaborate a bit about it.

Let’s start with explaining what can be seen: if a disk-assisted queue is configured, rsyslog will normally use the in-memory queue. It will open a disk queue only if there is good reason to do so (because it severely hurts performance). The prime reason to go to disk is when the in memory queue’s configured size has been exhausted. Once this happens, rsyslog begins to create spool files, numbered consequtively. This should normally happen in cases where e.g. a remote target is temporarily not reachable or a database engine is responding extremely slow.  Once the situation has been cleared, rsyslog clears out the spool files. However, one spool file always is kept until you shut rsyslog down. Note well that when the disk queue is idle, all messages are processed even though the the physical spool file still contains some already-processed data (impstats will show you the exact details).

This is expected behaviour, and there is good reason for it. This is what happens technically:

A DA queue is actually “two queues in one”: It is a regular in-memory queue, which has a (second) helper disk queue in case it neeeds it. As long as operations run smoothly, the disk queue is never used. When the system starts up, only the in-memory queue is started. Startup of the disk queue is deferred until it is actually needed (as in most cases it will never be needed).

When the in-memory queue runs out of space, it starts that Disk queue, which than allocates its queue file. In order to reclaim space, not a single file is written but a series of files, where old files are deleted when they are processed and new files are created on an as-needed basis. Initially, there is only one file, which is read and written. And if the queue is empty, this single file still exists, because it is the representation of a disk queue (like the in-memory mapping tables for memory queues always exist, which just cannot be seen by the user).

So what happens in the above case is that the disk queue is created, put into action (the part where it writes and deletes files) and is then becoming idle and empty. At that stage, it will keep its single queue file, which holds the queue’s necessary mappings.

One may now ask “why not shut down the disk queue if no longer needed”? The short answer is that we anticpiate that it will be re-used and thus we do not make the effort to shut it down and restart when the need again arises. Let me elaborate: experience tells that when a system needs the disk queue once, it is highly likely to need it again in the future. The reason for disk queues to kick into action are often cyclic, like schedule restarts of remote systems or database engines (as part of a backup process, for example). So we assume if we used it once, we will most probably need it again and so keep it ready. This also helps reduce potential message loss during the switchover process to disk – in extreme cases this can happen if there is high traffic load and slim in-memory queues (remember that starting up a disk queue needs comparativley long).

The longer answer is that in the past rsyslog tried to shut down disk queues to “save” ressources. Experience told us that this “saving” often resulted in resource wasting. Also, the need to synchronize disk queue shutdown with the main queue was highly complex (just think about cases where we shut it down at the same time the in-memory queue actually begins to need it again!). This caused quite some overhead even when the disk queue was not used (again, this is the case most of the time – if properly sized). An indication of this effect probably is that we could remove roughly 20% of rsyslog’s queue code when we removed the “disk queue shutdown” feature!

Bottom line: keeping the disk queue active once it has been activated is highly desirable and as such seeing a queue file around until shutdown is absolutely correct. Many users will even never see this spool file, because there are no exceptional circumstances that force the queue to go to disk. So they may be surprised if it happens every now and then.

Side-Note: if disk queue files are created without a traget going offline, one should consider increasing the in-memory queue size, as disk queues are obviouly much less efficient than memory queues.