Category: rsyslog

work log for rsyslog:

2007-09-26
– applied patch provided by varmojfekoj to support building ommysql
in its own way (now also resides in a plugin subdirectory)
– fixed a bug in cvthname() that lead to message loss if part
of the source hostname would have been dropped

Sorry folks, I didn’t manage to release 1.19.8 today. There was simply too much other activity that required attention. So I couldn’t finally decide on how to distribute from now on. Anyhow, it is my firm intention to release 1.19.8 tomorrow.

The good news, though, is that some folks are already testing that code. Maybe I get some feedback until tomorrow morning – and maybe that even helps me to get a better release…

Unfortunately, there is a serious bug in rsyslog 1.19.7 that can prevent UDP message reception. Totally. No message received at all. This somehow slipped through our own testing as well as test at some third parties.

The root cause of this problem is interesting: I changed an internal interface to make things more reliable. What I changed had some old code in them that did return’s right in the middle of the code. I overlooked those returns and so an invalid state was returned.

The interesting fact is that the function now returns an enum type (rsRetVal). Previously, it returned an integer. For some reason, the compiler issued no warning when the old (accidentally remaining) code returned an integer. If I’d receive a “wrong type” warning, I’d probably spotted it before even doing testing at all.

Bottom line: what’s wrong with my compiler settings?

Oh, and yes: I’ll release the fixed version soon. It then will be 1.19.8 to avoid confusion. I now fight with the distribution system: I received a patch that allows ommysql to be build as a separate module. So now it is separate – even from a distribution point of view. That causes some grief for distributing it and can possibly break some things in distribution packages. I need to think how to tackle that in the best possible way…

2007-09-25
– changed ttyname() to ttyname_r() – not a real fix, as this part of the
code was single threaded, but better to be prepared for the future.
– changed localtime() to localtime_r()
– released 1.19.7
– applied contributed patch to improve repeated message processing
(see https://bugzilla.redhat.com/show_bug.cgi?id=303341)

Hi all,

I’ve been a bit lazy reporting what I’ve done with rsyslog. The primary reason is that it was a lot of review, which is quite boring to report in a blog. Today, I think, I’ll do a new release, so at least here is my worklog for up until yesterday.

2007-09-17
– possibly found a bug in cvthname() that lead to a wrong size being specified
in a getnameinfo() API call – not sure, though, if it is “the” bug (actually,
it looks like it isn’t). – this is EXPERIMENTAL
– fixed a bug that caused signal handlers in cvthname() not to be restored when
a malicious pointer record was detected and processing of the message been
stopped for that reason (this should be really rare and can not be related
to the segfault bug we are hunting).
2007-09-18
– split the function cvthname() for clarity. Also changed to using the rsRetVal
status return system
– removed some compiler warnings in regard to signed / unsigned comparison
2007-09-20
– code cleanup
– fixed a minor memory leak that occured when the %APPNAME% property was
used (I think nobody used that in practice)
2007-09-21
– more review and cleanup
– simplified code in shouldProcessThisMessage() for debug output
– changed strerror() calls to thread-safe strerror_r() variant

I’ve not posted much the past days. That doesn’t mean nothing has happened. I wanted to post a work log today, but I have to admit I have forgotten it on a machine I right now can not access. OK, first thing tomorrow morning…

In short words, we are still on the bug hunt. I am now again back to reviewing code, this time on a functional basis. I am checking everything based on the message flow, looking at functions as they are called. Today, I completed the review of the reception part (up to the point when it gets into the main message queue). Unfortunately, no serious problem found. I used the review, however, to clean out some nits, add a large number of new comments and even found a memory leak. The later one would currently most probably never occur in practice, but when syslog-protocol gets adopted, it would have hit.

Tomorrow, and probably then next two or three days I’ll review the code that is executed once the message leaves to main queue. There is probably more meat for a bug in that part (its by far more complex).

I also think I’ll release the cleaned-up version sometime soon – after all, it’s better then what is currently released.

I keep you posted. Comments are always welcome.

As with a lot of open source projects, rsyslog funding is problematic. Of course, rsyslog currently is funded by my interest in it. And I am glad that Adiscon, my company, permits me to work on. The actual funding for all that action, kind of funny, mostly stems from closed source projects in the Windows world.

However, I’d like to see that over the years rsyslog can fund itself. In my point of view, funding should be provided by those that benefit most from it. So obviously, I do not expect any funding from private folks, people like you and me. Of course, everyone is invented to contribute and new code, doc and bug hunting is definitely a pro. On the other hand, companies (and other organizations, namely the government) take financial advantage of using it. Some of them also contribute in terms of time and code. This is great and much appreciated :)

For the others, I have begun to offer donations. Not surprisingly, this is not a real source of funding ;) But now I have taken an offer from sourceforge.net, which I think is interesting. They begin to offer a marketplace for open source services. So those folks that actually need help can be brought in contact with those that have experience. Sounds like a good idea to me and a fair way to fund projects.

I have created two test service offerings for rsyslog. One is to write a nice custom-created rsyslog configuration file. Well, with rsyslog’s relative ease of use, I do not expect that many folks use that. But you never know and I am always curios.

The other one is especially targeted towards organizations who must prove they have “official” support for all software they use (I guess this includes at least a number of government agencies). For them, I have created a email support option for rsyslog. It guarantees responses to support questions and this is often needed for auditing and other purposes. Of course, we still answer all support emails and do not plan to stop that.

I have added one real goody, though: and that is that we will provide patches for past releases of rsyslog. As you may know, developers hate to fumble with old releases. And so there is very little motivation to look at an older release when there is a new one out. With the paid support option, however, there is some motivation to do that. So, again, I think this is fair: we are just offering a service that would otherwise never appear. It doesn’t hurt any of the other users.

I am very interested to see how this works out. I would also be interested in feedback from the field. How do you like this idea? Do you have any other/additional suggestions?

Again, I am doing small changes, mostly review, at this time. So I have batched up some things. Probably I’ll switch back to a more daily mode, as it allows to keep better in touch. We’ll see (comments appreciated).

Here comes the log:

2007-09-10
– bumped version number to 1.19.6
– fixed a bug that in –enable-debug mode caused an assertion when the
discard action was used
2007-09-07
– added eCmdHdlrGetWord command handler
– added $ModDir config directive
– modified $ModLoad so that an absolute path may be specified as
module name (e.g. /rsyslog/ommysql.so)
– applied patch by varmojfekoj two fix two potential segfault situations
– cleaned up some signed/unsinged char issues
– released 1.19.5
2007-09-10
– bumped version number to 1.19.6
– fixed a bug that in –enable-debug mode caused an assertion when the
discard action was used
2007-09-11
– applied patch by varmojfekoj to change signal handling to the new
sigaction API set (replacing the depreciated signal() calls and its
friends.
– did a full review of all remaining TODO items in code – nothing of
importance found (but some minor nits like comments fixed)
– cleaned up compiler warnings
– applied patch by varmojfekoj to FIX a bug that could cause
segfaults if empty properties were processed using modifying
options (e.g. space-cc, drop-cc)
– fixed man bug: rsyslogd supports -l option
– checked -s/-l option and found that they work as expected – closed case
– added some comments in relation to -s/-l option
– released 1.19.6
– removed compiler warnings in non-debug mode
– fixed a bug that caused the CStr class to not honor the parameter to
return NULL on empty string – causes a mem leak and can create favourable
environment for other bugs (as it leads to empty hostnames)
– pulled 1.19.6 (no downloads so far), created new 1.19.6 with the bugfix,
re-released that (now tagged v1-19-6b in cvs)

We have received feedback that the 1.19.x releases of rsyslog contain a bug that leads to a segfault after some time. Of course, this is a very bad thing to happen. After all, the primary goal for the rsyslog project is reliability and it should be able to survive even in the worst conditions (e.g. low system memory). So this bug is clearly unacceptable and has received highest priority.

Unfortunately, it is hiding very well. We are all looking into troubleshooting this beast. Thankfully, there is a lot of community support, especially with testing. What makes this bug so hard to find is, among others, the inability to reproduce it in lab. So the project is totally dependent on user feedback. As current reports have shown, we also currently assume that releases prior to 1.19.0 can also be affected – so it is not as easy as checking the changes.

I receive very valuable help by varmojfekoj. He is already a frequent and great contributor of patches. His skills and experience are extremely valuable and I am very glad to have him work on this project. Please give him a big hand. Thanks, varmojfekoj, the project would not be that far without your help!

While the bug itself is obviously a very bad thing, it has some good side-effects. Most importantly, the code is getting another round of very in-depth review. That review has already brought up some fixes for obscure situations. Situations that I’d expect to happen extremely infrequently (if at all) in reality. But now even those are fixed. And I am sure that the review will bring even more benefit as it continues. Of course, I’d be glad if we find the bug ASAP, but it is good to know that all that work also provides additional benefit (at least that keeps my depression level low ;)).

The bad thing, besides the bug itself, is that the bug hunt obviously defers other work. Most importantly, I refrain from making any changes that may not be related to clean up or bug fixing. I think it is not smart to introduce new code (read: complexity) at this stage of the project. And, of course, all resources should focus on fixing that bug, so there wouldn’t even be time to enhance other parts of the code. So for all of you waiting for new features: please bear with us, the implementation schedule will slip a little bit.

If you experience stability problems, please report. Each reports helps us understand the problem somewhat better.

Now that I am back working on the rsyslog code, I will continue to provide information about what I am doing. In most cases, however, I will not provide daily logs. The reason is that I currently focus on new design and will leave rsyslog code mature. So changes will be relatively infrequent (at least I plan so ;]).

Here is the work log for the past days:

2007-09-04
– integrated patches from Michel Samia and varmojfekoj
– released 1.19.4
– changed some calls to CStr class to their “safe” counterpart – they could
cause program aborts if the object in question was an empty string
2007-09-05
– added some links to doc
– removed an invalid config sample from sample.conf
– changed part of the CStr interface so that better error tracking
is provided and the calling sequence is more intuitive (there were
invalid calls based on a too-weired interface)
– (hopefully) fixed some remaining bugs rooted in wrong use of
the CStr class. These could lead to program abort.
2007-09-06
– added forwarding information to sysklogd (requires special template)
to config doc