Category: rsyslog

Finally, rsyslog v4 is materializing. Yesterday, I released the first devel version that is named 4.1.0. This starts a totally new branch. I decided to finally move on to v4 because I am enhancing performance quite a bit and this causes a number of big changes to the core engine as well as many modules. So rather than doing all of this in v3, I thought it is a good time to move to a new major version.

I expect that the new code will de-stabilize the project for some time and so I have now a feature-rich v3-stable release which will be available to everyone, at the price of less performance. That doesn’t mean v3 is slow, but v4 is even much faster. So I can finally begin to experiment a bit more with the new v4 branch and don’t need to think too hard that I may be introducing changes that are hard to roll into a stable within a reasonable time frame.

As far as the first v4-stable is concerned, I do not expect one to surface before February 2009 and, obviously, it will be not as stable as v3-stable is.

So the game is starting once again, and I hope you enjoy it ;)

Regular expressions are quite powerful, but the syntax in rsyslog is, well, not easy to use. Also, as we have seen, the usual regex check tools don’t work always well with rsyslog’s POSIX expressions. I have created a web-based regular expression checker/generator today. It is more or less finished, but of course needs fine-tuning.

If this tool turns out to be useful (judging on comments and access count over the next weeks), I will probably do some other online tools aiding in config generation. This will be part of the overall effort to make it easier to unleash rsyslog’s full potential (all too often people simply do not know what magic they could do ;)).

There has been some discussions about rsyslog HUP processing. Traditionally, SIGHUP is used to signal the syslogd to a) close its files and b) reload its config. Rsyslog carried over this behavior from sysklogd.

However, rsyslog is much more capable than sysklogd. Among others, it is able to buffer messages that were received, but could not yet be processed. To remain compatible to the sysklogd of doing HUP, rsyslogd does a full daemon restart when it is HUPed. Among others, that means that messages from the queue are discarded, at least if the queue is configured with default settings. David Lang correctly stated that this may surprise some, if not most users. While I am still of the view that discarding the queue, under these circumstances, is the right thing to do, I agree it may be surprising (I added a hint to the man pages recently to reduce the level of surprise).

Still, there is no real need to do a full daemon restart in most cases. The typical HUP case is when logrotation wants to rotate files away and it needs to tell rsyslogd to close them. Actually, I asked if anybody knew any script that HUPs rsyslog to do a full config reload. The outcome was that nobody knew. However, some people liked to stick with the old semantics, and there may be reason to do so.

I have now implemented a lightweight HUP to address this issue. It is triggered via a new configuration directive, $HUPisRestart. If set to “on”, rsyslogd will work as usual and do a (very, very expensive) full restart. This is the default to keep folks happy that want to keep things as backwards-compatible as possible. Still, I guess most folks will set it to “off”, which is the new non-restart mode. In it, only output files are closed. Actually, the output plugin receive a HUP notification and can do whatever it likes. Currently, onle omfile acts on that and closes any open files. I can envision that other outputs, e.g. omfwd, can also be configured to do some light HUP action (for example close outbound connections).

The administrator needs to select either mode for the system. I think this is no issue at all and it safes me the trouble to define multiple signals just to do different types of HUP. My suggestion obviously is to use the new lightweight HUP for file closing, which means you need not to change anything for logrotate et al. Then, when you need to do a config reload, do a “real” restart by issuing a command like “/etc/init.d/rsyslogd restart”. And if there really exists a script that requires a config-reload HUP, that should be changed accordingly.

Finally, it is time to think about the next major rsyslog release! I have done many enhancements in v3, but the latest performance optimization work leads to a couple of significant changes in the core engine. I think it makes sense to roll these into a new major release. That leaves folks with the option to keep at the feature-rich v3-stable branch, while avoiding some of the potential unavoidable bugs in the upcoming v4 branch.

From a feature point of few, version 3 would have been good for at least three to four major releases, which I did not do just because to prevent you from coming scared by the pace with which we are moving ;)- so I think it now is a perfect spot to begin developing v4. I hope that we will see a first beta of that branch around xmas, which, I think, is a nice gift.

Some of you may know that I am a native German speaker. I thought I started an interesting experiment: the “deutsches rsyslog forum” (which means “german-language rsyslog forum” ;)). It is targeted to those who prefer to express themselves in German language.

The interesting question, though, is if this forum will actually attract much attention. In German IT, there is a tendency to think that almost everyone speaks sufficiently well English so that he or she can obtain enough information to get the job done. If that proves true, there would be very little benefit in localizing any of the documentation into German language. So before seriously considering that, it is probably a good idea to do some testing. For the very same reason, my buddy Tom Bergfeld currently translates the rsyslog home page, mainly the announcements, into German. We will do a similar experiment to phpLogCon and evaluate both together after some time has progressed.

Please drop me a note if you have an opinion on this, or on localization at all.

Thanks,
Rainer

I have not blogged that much the past weeks. Rsyslog work is still progressing quite nicely. I am currently working on (large) performance enhancements. Thanks to David Lang for his help on this topic.

I am also hunting another threading bug. This one manifests only when running on high-end hardware and seems to have to do with synchronization. Interestingly, the problem does not show up under valgrind (a memory and threading debugger), which usually points to flaws very well. Hard to find… Especially hard to find because I do not have the right hardware to reproduce it. I managed to get a faster box last week, but still it is not fast enough (and does not have enough cores…) to reliably trigger the problem. Since them, I have seen it a couple of times, but no indication yet of what is going on.

Also, I need to help some other Adiscon projects and do a bit of my consulting chores, so that the time allocated to rsyslog is a bit limited. But, hey, I have worked nearly full time on it this year and it has evolved very well. So I don’t think it is a problem going at a somewhat lower pace for the time being (plus, of course, I hope some other sources of funding will appear which enable me to go back to “full time rsyslog” mode).

In any case, the future has exciting things to come up with. I personally would like to see a customizable message parser, which would enable users to work with different sender formats at the same time.