Category: rsyslog

I posted an outline of my next actions on the rsyslog mailing list and would like to share it here as well:

I have thought about setting up a full lab for GSS-API before carrying on. For now, I have decided to NOT do that. I am sure that the contributors have tested it quite well and the code that I have reviewed looks excellent.

So I will pull it in as is and wait for some feedback from the field (with the assumption “no feedback” equals “OK”).

I will then begin to look at the loadable module de-initialization. This is not really clean in the current release, but that’s no problem because modules never get unloaded. However, in the long term we need this to be clean.

The mysterious segfault issue is still dangling. I was hesitant to do any larger-scale new development without fixing it. But given the fact that it is extremely hard to find, and obviously happens very seldom, I’ll continue developing. I am right now looking into upgrading the dev machine to an x64 OS, where most of the problems happened. My hope is that I will see a segfault during further development work and then hopefully be able to tackle it. I still think that the segfault must be well understood and fixed before I go into some serious multithreading redesign. As such, unfortunately, this issue still holds some of the work scheduled for the next *major* version.

I thought I give you an update here in my end (will also post this to the blog for the others). Any feedback/suggestion is highly welcome.

I copy over some usage notes from Peter Vrabec’s intial announcement of the patch:

It adds a new commandline option ‘-g’ to listen for a connection wrapped with gss-api and few new configuration directives:

for server:

$gsslistenservicename <service name>

for client:

$gssforwardservicename <service name>
$gssmode <encryption|integrity|none>

With gssmode set to “encryption” or “integrity” all tcp selectors will be forwarding messages via gss-api.

That’s probably useful while I am getting up some real documentation.

I am now back in Germany for a week. I’ve now managed to get through all these nice “gifts” that were waiting for me while I was in Florida. Today, I think, I’ll be able to resume work on rsyslog. First on my todo list will be solving a problem with ommysql (or, better said, providing some instrumentation to get hold of it) as well as integrating a very interesting patch sent over from the red hat folks. So … stay tuned and expect updates to this blog to happen once again.

A note to all fellow rsyslog users and contributors. I got a chance to view space shuttle Discovery’s STS-120 launch next week. I will fly over to Florida tomorrow. I will be available via email, but my response time will be sluggish. My buddy Michael will handle things related to rsyslog in the mean time. Most importantly, he’ll release 1.19.10 tomorrow while I am on the plane. Please keep thoughts and contributions flowing, they are very welcome.

A couple of changes the last days. They are all now in cvs. Tomorrow we will release 1.19.10.

2007-10-15
– bumped version number
– changed ommsyql to allow for “:ommysql:” module specific action call method
instead of “>”. This shall facilitate the creation of other plugins.
– added notes about new action module designator to module documentation
– updated user doc to new module action calling convention
2007-10-16
– updated rsyslogd doc set man page; now in html format
2007-10-17
– undid creation of a separate thread for the main loop — this did not
turn out to be needed or useful, so reduce complexity once again.
– added doc fixes provided by Michael Biebl – thanks

I have just released rsyslog 1.19.9. It is now two weeks since the last release. I have taken some extra time to make sure that the release system (source tarball) now fits the packagers’ and user’s needs. Also, some time went into hunting the segfault, though this was still quite fruitless. However, mildew has identified that the segfault seems to occur only on 64 bit machines, which is a very good hint.

I’ll now watch for comments on the 1.19.9 release and then see if I myself can do anything against the segfault early next week (I am setting up another x64 machine for testing). Later the week, I’ll be heading for my space shuttle launch viewing trip, which will unfortunately mean that my focus will not be on rsyslog (by my co-workers will keep it up).

Work on rsyslog was focussed on the new packaging and bug fixing. Most importantly, people had problems with the new tarball introduced in 1.19.8 and we have made sure that 1.19.9 will again meet the expectations. We’ve done a round of pre-releases on the mailing list and managed to get down to something that seems to be quite acceptable. The official 1.19.9 release will most probably be released later today.

Here are the details:

2007-10-08
– changed the threading to include one extra thread that runs the
mainloop. This was done to work around a problem with malloc/free. Note
that we are still running on two threads – the startup thread just waits
for the new one to finish. For a description of the problem I try to
work-around, please see:
http://rgerhards.blogspot.com/2007/10/could-i-really-reproduce-bug.html
Note that this is an experimental change, which will only stay if it
proves to fix the segfault issue we are dealing with.
2007-10-10
– changed the build system to use a single source tarball again (but
different makefiles for the main project and ommysql)
thanks to varmojfekoj for the patch
2007-10-11
– applied fixes from Michael Biebl:
1.) fix failing compilation of ommysql plugin (s/ommysql-config.h/config.h/)
2.) fix mysql configure check (although the default is no, we did
check for the mysql devel files)
3.) Create a separate Makefile.am for the doc files. This cleans up
the toplevel Makefile.am considerably and makes it much more readable
and maintainable.
3b) Assign the html doc files to html_DATA. This means, they are
installed to $(hmtdir), which by autoconf standards is
$(prefix)/share/doc/$packagename/.
4.) Reformat the SOURCES line to make it better readable and maintainable.
– integrated patch by varmojfekoj to tweak build system even further
– Peter Vrabec requested doc not to be “stored” in html_DATA – changed that

Peter Vrabec provided very helpful information on the anomaly I experienced with malloc/free under mudflap instrumentation. See his report:


$ gcc -lmudflapth -lpthread -fmudflapth mud.c
.........----------
mud.c: In function ‘main’:
mud.c:27: warning: return type of ‘main’ is not ‘int’
./a.out
alloc p in thread: 0ea72530
alloc p in main thread: 0ea72460
freeing p from thread: 0ea72530
free done!
freeing p from main thread: 0ea72460
free done!
main thread exiting

gcc -lpthread -fmudflapth mud.c -lmudflapth
................................----------
mud.c: In function ‘main’:
mud.c:27: warning: return type of ‘main’ is not ‘int’
$ ./a.out
alloc p in thread: 1bffe3f0
alloc p in main thread: 1bffe440
freeing p from thread: 1bffe3f0
free done!
freeing p from main thread: 1bffe440
*** glibc detected *** ./a.out: double free or corruption (out):
0x000000001bffe440 ***
======= Backtrace: =========
/lib64/libc.so.6[0x32bde6e8a0]
/lib64/libc.so.6(cfree+0x8c)[0x32bde71fbc]
./a.out(__wrap_main+0x174)[0x400924]
/lib64/libpthread.so.0[0x32bea061b5]
/lib64/libc.so.6(clone+0x6d)[0x32bdecd39d]
======= Memory map: ========
bla bla bla

Note the position of the -lmudflapth argument. So, as it looks, the problem was really one of the instrumentation and not of rsyslogd itself. So, bad as it is, we are still back to hunting a bug that is hiding well. But hopefully we’ll get somewhat closer when mudflap is now actually active… I’ll see and post news as soon as I have them.

I have now created a work-around for the malloc/free threading issue which creates an extra thread for the mainloop. The startup thread then simply waits for it to be finished. As such, there is never memory allocated in the “main” thread. At least for me, it works now with mudflap. I am still in doubt if that was the segfault issue (or just a bug in mudflap), but at least we can give it a try.

I will now see that I get some feedback. The next thing is change the packaging back to a single source tarball (by popular request ;)).