Author: Rainer Gerhards

Yesterday’s rsyslog work log:

2008-01-08
– fixed doc bug — thanks to varmojfekoj for pointing it out
– fixed some memory leaks in new code — thanks to varmojfekoj for the patch
– implemented queue disk reader to switch to multiple files
– first implementation of “disk” queue mode finished. It still needs some
work and the deserializer needs also to be expanded, but the queue at
least performs well now.
– fixed a race condition that could occur when input modules were terminated
– added –enable-mudflap ./configure option
– completed deserialization support in msg object (but not deserializer itself)
– completed object deserializer

rsyslog work log. I am currently totally focused on the queue (and thus not very responsive on mail and otherwise):

2008-01-07
– released 3.10.0
– performance-tuned stringbuf class
– implemented disk queue as far as I could without an object de-serializer
– implemented buffered read calls for the queue file
– implemented class type registry
– MsgSetProperty() implemented
– defined a property class
– implemented deserializer (needs some more work)

Here is the rsyslog work log for yesterday:

2008-01-06
– fixed a bug with integer conversion in srUtils.c
– changed some lib functions to work on long instead of int
to care for 64 bit platforms (just to be on the save side)
– worked a bit on object serialization
– cleaned up msg structure (interestingly, there were for example
two fields with identical meaning and iSyslogVersion was never
used ;))
– completed serializer for msg (but needs review)
– did a little bit performance cleanup
– worked on object header (now also contains the size)

If you followed my work logs or CVS updates, you’ve probably seen that I have worked quite a bit on rsyslog‘s threading. So I thought I share a view “design documents” that cover up the big picture.

Michael Biebel asked me for a few graphical representations of how the modules interact and what the message flow is. I am not a real good computer graphics guy, and an old-fashioned one. So I thought before I let you wait any longer, I share some of my hand sketches. They are not fancy, probably hard to read – but maybe still helpful. Find them below. A klick bring up the hires version, which is a bit less hard to read ;)

I’ll try to add better graphics and descriptions as soon as I find some time. But I have to admit that I currently have so many things on my mind that I’d like to code first. So it may take a short while.

Here is the rsyslog work log for the past days:

2008-01-03
– fixed a few typos noticed by Jonathan Smith – thanks
– moved queue code to its own module (finally)
– restructured queue interface to use rsRetVal and instances, removed
dependency on globals – now more like a real class
– implemented queue type “drivers”
– queue is now a full object and handles threading by itself
– applied Michael Biebl’s patch to clean up the makefiles
– added capability to use a linked list for queuing to the queue class
– added $MainMsgQueueType config parameter
– some cleanup
– added $SpoolDirectory config parameter
– added $MainMsgQueueFilePrefix config parameter
– begun working on disk queueing (not completed, do not use this mode!)
– begun some work on Msg Object serializiation
2008-01-04
– created a kind of general base class
– removed serialization pointer from queue; used new base class instead
– utilized the new auto-destruction capability so that the queue can now
destruct user objects if needed
– changed queue object Construction/Startup interface
2007-08-05
– added capability for concurrent access to the msg class. Can be dynamically
activated. If active, locking is employed.
– added the “direct” queueing mode to queue class (no queing at all)
– added multiple worker thread capability to queue class
– implemented $MainMsgQueueWorkerThreads config directive
– removed some no-longer-needed code (thanks Michael Biebl for the help)

A short rsyslog work log form the past days:

2007-12-31
– created omtesting, a debug and development aid output module. This is stage
work for the new queueing engine – we need a way to delay rule execution
and that’s what the module currently does ;)
2008-01-02
– released 2.0.0

Hi folks, probably the last rsyslog work log post for 2007. Thanks for sticking around – and hopefully I’ll see you again in 2008. It’ll become a very exciting year, with a lot of new features. I am eager to implement what is right now on my head, and I’ll most probably will start with modifying the message queue, an endeavor that will ultimately lead to store-and-forward capability just like in syslog-ng’s premium edition. And the good news is that I hope to finish that in January 2008 ;) — what also means that I have made up my priorities. Was not an easy job, and I hope I got it right. So store-and-forward with enhanced output threading is first and the other things will follow later. To me, the hardest decision was to put off expressions, another feature at least I would like to see the sooner the better.

But now back to the work log:
2007-12-27
– added $UDPServerAddress config directive
– added capability to have multiple UDP listeners running concurrently
– applied cross-platform patch from darix to facilitate GSS-API compile
on more platforms
– some cleanup
– internal restructuring in omfwd.c – stage work for further modularization
I think I also fixed a bug as a side-effect – but not looked to much at it
2007-12-28
– took TCPSend() apart and made it generic via function pointers
– moved TCPSend() and frame building code to tcpsyslog.c
– omgssapi created
– removed gss-api code from omfwd.c