Category: rsyslog

rsyslog work log. I am currently totally focused on the queue (and thus not very responsive on mail and otherwise):

2008-01-07
– released 3.10.0
– performance-tuned stringbuf class
– implemented disk queue as far as I could without an object de-serializer
– implemented buffered read calls for the queue file
– implemented class type registry
– MsgSetProperty() implemented
– defined a property class
– implemented deserializer (needs some more work)

Here is the rsyslog work log for yesterday:

2008-01-06
– fixed a bug with integer conversion in srUtils.c
– changed some lib functions to work on long instead of int
to care for 64 bit platforms (just to be on the save side)
– worked a bit on object serialization
– cleaned up msg structure (interestingly, there were for example
two fields with identical meaning and iSyslogVersion was never
used ;))
– completed serializer for msg (but needs review)
– did a little bit performance cleanup
– worked on object header (now also contains the size)

If you followed my work logs or CVS updates, you’ve probably seen that I have worked quite a bit on rsyslog‘s threading. So I thought I share a view “design documents” that cover up the big picture.

Michael Biebel asked me for a few graphical representations of how the modules interact and what the message flow is. I am not a real good computer graphics guy, and an old-fashioned one. So I thought before I let you wait any longer, I share some of my hand sketches. They are not fancy, probably hard to read – but maybe still helpful. Find them below. A klick bring up the hires version, which is a bit less hard to read ;)

I’ll try to add better graphics and descriptions as soon as I find some time. But I have to admit that I currently have so many things on my mind that I’d like to code first. So it may take a short while.

Here is the rsyslog work log for the past days:

2008-01-03
– fixed a few typos noticed by Jonathan Smith – thanks
– moved queue code to its own module (finally)
– restructured queue interface to use rsRetVal and instances, removed
dependency on globals – now more like a real class
– implemented queue type “drivers”
– queue is now a full object and handles threading by itself
– applied Michael Biebl’s patch to clean up the makefiles
– added capability to use a linked list for queuing to the queue class
– added $MainMsgQueueType config parameter
– some cleanup
– added $SpoolDirectory config parameter
– added $MainMsgQueueFilePrefix config parameter
– begun working on disk queueing (not completed, do not use this mode!)
– begun some work on Msg Object serializiation
2008-01-04
– created a kind of general base class
– removed serialization pointer from queue; used new base class instead
– utilized the new auto-destruction capability so that the queue can now
destruct user objects if needed
– changed queue object Construction/Startup interface
2007-08-05
– added capability for concurrent access to the msg class. Can be dynamically
activated. If active, locking is employed.
– added the “direct” queueing mode to queue class (no queing at all)
– added multiple worker thread capability to queue class
– implemented $MainMsgQueueWorkerThreads config directive
– removed some no-longer-needed code (thanks Michael Biebl for the help)

A short rsyslog work log form the past days:

2007-12-31
– created omtesting, a debug and development aid output module. This is stage
work for the new queueing engine – we need a way to delay rule execution
and that’s what the module currently does ;)
2008-01-02
– released 2.0.0

Hi folks, probably the last rsyslog work log post for 2007. Thanks for sticking around – and hopefully I’ll see you again in 2008. It’ll become a very exciting year, with a lot of new features. I am eager to implement what is right now on my head, and I’ll most probably will start with modifying the message queue, an endeavor that will ultimately lead to store-and-forward capability just like in syslog-ng’s premium edition. And the good news is that I hope to finish that in January 2008 ;) — what also means that I have made up my priorities. Was not an easy job, and I hope I got it right. So store-and-forward with enhanced output threading is first and the other things will follow later. To me, the hardest decision was to put off expressions, another feature at least I would like to see the sooner the better.

But now back to the work log:
2007-12-27
– added $UDPServerAddress config directive
– added capability to have multiple UDP listeners running concurrently
– applied cross-platform patch from darix to facilitate GSS-API compile
on more platforms
– some cleanup
– internal restructuring in omfwd.c – stage work for further modularization
I think I also fixed a bug as a side-effect – but not looked to much at it
2007-12-28
– took TCPSend() apart and made it generic via function pointers
– moved TCPSend() and frame building code to tcpsyslog.c
– omgssapi created
– removed gss-api code from omfwd.c

I initially sent this message only to the mailing list. But now I think it make sense to reproduce it here. So there we go:

I am working on the modular structure of rsyslog v3. I am currently revisiting gss-api support. I notice that with the current omfwd, it will be extremely hard to separate gss-api support into its own module. Doing so will break backward compatibility to the configuration file.

GSS-API has been out only for a few days, and mostly over the holiday period. So it is much less of a concern if we introduce now some changes that will case rsyslog.conf format modifications. Much less trouble than when we release v2, a release expected to be in wide use for at least half a year, if not much longer. V2 released with the current syntax would require me to do some tricks in v3 to keep compatibility. Quite complex.

So I decided to create a omgssapi for v3 and extract the gss-api code from omfwd. It looks like this can be done without too much code duplication. There will be some duplicate code, but it will shrink as v3 continues to be developed. Once I have a good working version, which I expect very soon, I will backport that to the v1/2 source tree. I’ll then do a new v1 release with a slightly incompatible gss-api config file syntax. After this is out for a few days, I hope I can than finally push out that version as v2.

I hope this is a good decision. I think it will save us major future trouble at the expense of a relatively slight disturbance in the late v1 timeline. I guess most user’s won’t even notice there is a change.

As always, Feedback is appreciated.

I have made good progress with rsyslog‘s input modules. As it looks, the basic things are done and the input module interface has been proven to be both quite stable as well as very simple. It doesn’t yet support different instances, but I begin to think that I do not even need them – also not in the long term.

Of course, most of the current input modules are not clean modules. They have a lot of dependencies to other parts of the code, which is not yet able to be dynamically loaded. But at least there is a foundation on which additional modules could be build. Getting the current input modules to be real clean modules will require further stage work. Many thanks need to be done.

So what to do next? It now comes down to both a matter of priorities and dependencies. I am writing this note here mostly for myself. It helps my clear up my thoughts and will also probably serve as a reference for quite a while. My thoughts may be hard to understand – sorry for that. But I thought I make them public when I write them down – even if they are not really targeted toward others. I still hope they may help you get some more background info.

So what’s to do:

- find a way to handle global settings
- multi-threaded output modules
  a perquisite for
  - create queued outputs (write to queue if action fails and 
    restart when it resumed)
- re-write way config file is read
  probably perquisite for:
  - create expression support
    - in templates
    - in selector filters
- create interface for (loadable) user function modules
- create a system to allow loading "library" loadable modules
  (e.g. network library for imudp, imtcp, ...)
  - separate GSSAPI from plain TCP (requires libs and lib extension system)

I’ve been a bit busy with rsyslog over the xmas break. Here are the changes:

2007-12-25
– moved some more net functionality out of syslogd.c – stage work
– fixed duplicate license text in syslogd.c – made it ambigious
– moved udp net code – again, stage work
– moved some of the udp input code to its right place
2007-12-26
– moved cross-platform define for AI_NUMERICSERV to net.h
– made udp code somewhat less dependable on global variables — stage work
– removed omfwd code dependency on “finet”
– removed imudp code dependency on “finet”
– removed active INET code from syslogd.c – still some auxiliary things
remain
– fixed socket leak in omfwd.c
– removed global variable LogPort
– removed gloable variable AcceptRemote and external def of bFinished