Category: rsyslog

rsyslog work log for Jan, 9th and 10th (sorry, forgot to post yesterday):

2008-01-09
– implemented new GetSize() handler for config files
– implemented $MainMsgQueueMaxFileSize configuration directive
– implemented queue object method to set the file name prefix
– implemented $MainMsgQueueFilePrefix configuration directive
– created a generic stream class (for file access)
– changed queue class to use stream class
– some cleanup on object model
2008-01-10
– changed queue file name generation (to be more generic)
– made queue file names better readable
– added buffered output to stream class
– data record support added to stream output writer
– added write functions for several types to stream class
– changed objSerialize methods to work directly on the stream class
– prepared for 3.10.1 release
– changed some config parameters and some cleanup
– released 3.10.1
– fixed a bug that caused a segfault on startup when no $WorkDir directive
was specified in rsyslog.conf
– fixed a bug that caused a segfault on queues with types other than “disk”
– removed the no longer needed thread TermSyncTool
– re-released 3.10.1
– implemented strm object serializer (untested as the code required for test
is not yet present – hen/egg problem…)
– some cleanup
– implemented management function for worker thread 0 in order to change
queue workers dynamically — stage work

Yesterday’s rsyslog work log:

2008-01-08
– fixed doc bug — thanks to varmojfekoj for pointing it out
– fixed some memory leaks in new code — thanks to varmojfekoj for the patch
– implemented queue disk reader to switch to multiple files
– first implementation of “disk” queue mode finished. It still needs some
work and the deserializer needs also to be expanded, but the queue at
least performs well now.
– fixed a race condition that could occur when input modules were terminated
– added –enable-mudflap ./configure option
– completed deserialization support in msg object (but not deserializer itself)
– completed object deserializer

rsyslog work log. I am currently totally focused on the queue (and thus not very responsive on mail and otherwise):

2008-01-07
– released 3.10.0
– performance-tuned stringbuf class
– implemented disk queue as far as I could without an object de-serializer
– implemented buffered read calls for the queue file
– implemented class type registry
– MsgSetProperty() implemented
– defined a property class
– implemented deserializer (needs some more work)

Here is the rsyslog work log for yesterday:

2008-01-06
– fixed a bug with integer conversion in srUtils.c
– changed some lib functions to work on long instead of int
to care for 64 bit platforms (just to be on the save side)
– worked a bit on object serialization
– cleaned up msg structure (interestingly, there were for example
two fields with identical meaning and iSyslogVersion was never
used ;))
– completed serializer for msg (but needs review)
– did a little bit performance cleanup
– worked on object header (now also contains the size)

If you followed my work logs or CVS updates, you’ve probably seen that I have worked quite a bit on rsyslog‘s threading. So I thought I share a view “design documents” that cover up the big picture.

Michael Biebel asked me for a few graphical representations of how the modules interact and what the message flow is. I am not a real good computer graphics guy, and an old-fashioned one. So I thought before I let you wait any longer, I share some of my hand sketches. They are not fancy, probably hard to read – but maybe still helpful. Find them below. A klick bring up the hires version, which is a bit less hard to read ;)

I’ll try to add better graphics and descriptions as soon as I find some time. But I have to admit that I currently have so many things on my mind that I’d like to code first. So it may take a short while.

Here is the rsyslog work log for the past days:

2008-01-03
– fixed a few typos noticed by Jonathan Smith – thanks
– moved queue code to its own module (finally)
– restructured queue interface to use rsRetVal and instances, removed
dependency on globals – now more like a real class
– implemented queue type “drivers”
– queue is now a full object and handles threading by itself
– applied Michael Biebl’s patch to clean up the makefiles
– added capability to use a linked list for queuing to the queue class
– added $MainMsgQueueType config parameter
– some cleanup
– added $SpoolDirectory config parameter
– added $MainMsgQueueFilePrefix config parameter
– begun working on disk queueing (not completed, do not use this mode!)
– begun some work on Msg Object serializiation
2008-01-04
– created a kind of general base class
– removed serialization pointer from queue; used new base class instead
– utilized the new auto-destruction capability so that the queue can now
destruct user objects if needed
– changed queue object Construction/Startup interface
2007-08-05
– added capability for concurrent access to the msg class. Can be dynamically
activated. If active, locking is employed.
– added the “direct” queueing mode to queue class (no queing at all)
– added multiple worker thread capability to queue class
– implemented $MainMsgQueueWorkerThreads config directive
– removed some no-longer-needed code (thanks Michael Biebl for the help)

A short rsyslog work log form the past days:

2007-12-31
– created omtesting, a debug and development aid output module. This is stage
work for the new queueing engine – we need a way to delay rule execution
and that’s what the module currently does ;)
2008-01-02
– released 2.0.0

Hi folks, probably the last rsyslog work log post for 2007. Thanks for sticking around – and hopefully I’ll see you again in 2008. It’ll become a very exciting year, with a lot of new features. I am eager to implement what is right now on my head, and I’ll most probably will start with modifying the message queue, an endeavor that will ultimately lead to store-and-forward capability just like in syslog-ng’s premium edition. And the good news is that I hope to finish that in January 2008 ;) — what also means that I have made up my priorities. Was not an easy job, and I hope I got it right. So store-and-forward with enhanced output threading is first and the other things will follow later. To me, the hardest decision was to put off expressions, another feature at least I would like to see the sooner the better.

But now back to the work log:
2007-12-27
– added $UDPServerAddress config directive
– added capability to have multiple UDP listeners running concurrently
– applied cross-platform patch from darix to facilitate GSS-API compile
on more platforms
– some cleanup
– internal restructuring in omfwd.c – stage work for further modularization
I think I also fixed a bug as a side-effect – but not looked to much at it
2007-12-28
– took TCPSend() apart and made it generic via function pointers
– moved TCPSend() and frame building code to tcpsyslog.c
– omgssapi created
– removed gss-api code from omfwd.c

I initially sent this message only to the mailing list. But now I think it make sense to reproduce it here. So there we go:

I am working on the modular structure of rsyslog v3. I am currently revisiting gss-api support. I notice that with the current omfwd, it will be extremely hard to separate gss-api support into its own module. Doing so will break backward compatibility to the configuration file.

GSS-API has been out only for a few days, and mostly over the holiday period. So it is much less of a concern if we introduce now some changes that will case rsyslog.conf format modifications. Much less trouble than when we release v2, a release expected to be in wide use for at least half a year, if not much longer. V2 released with the current syntax would require me to do some tricks in v3 to keep compatibility. Quite complex.

So I decided to create a omgssapi for v3 and extract the gss-api code from omfwd. It looks like this can be done without too much code duplication. There will be some duplicate code, but it will shrink as v3 continues to be developed. Once I have a good working version, which I expect very soon, I will backport that to the v1/2 source tree. I’ll then do a new v1 release with a slightly incompatible gss-api config file syntax. After this is out for a few days, I hope I can than finally push out that version as v2.

I hope this is a good decision. I think it will save us major future trouble at the expense of a relatively slight disturbance in the late v1 timeline. I guess most user’s won’t even notice there is a change.

As always, Feedback is appreciated.

I have made good progress with rsyslog‘s input modules. As it looks, the basic things are done and the input module interface has been proven to be both quite stable as well as very simple. It doesn’t yet support different instances, but I begin to think that I do not even need them – also not in the long term.

Of course, most of the current input modules are not clean modules. They have a lot of dependencies to other parts of the code, which is not yet able to be dynamically loaded. But at least there is a foundation on which additional modules could be build. Getting the current input modules to be real clean modules will require further stage work. Many thanks need to be done.

So what to do next? It now comes down to both a matter of priorities and dependencies. I am writing this note here mostly for myself. It helps my clear up my thoughts and will also probably serve as a reference for quite a while. My thoughts may be hard to understand – sorry for that. But I thought I make them public when I write them down – even if they are not really targeted toward others. I still hope they may help you get some more background info.

So what’s to do:

- find a way to handle global settings
- multi-threaded output modules
  a perquisite for
  - create queued outputs (write to queue if action fails and 
    restart when it resumed)
- re-write way config file is read
  probably perquisite for:
  - create expression support
    - in templates
    - in selector filters
- create interface for (loadable) user function modules
- create a system to allow loading "library" loadable modules
  (e.g. network library for imudp, imtcp, ...)
  - separate GSSAPI from plain TCP (requires libs and lib extension system)