Category: rsyslog

Here is the rsyslog work log for the past days:

2008-01-03
– fixed a few typos noticed by Jonathan Smith – thanks
– moved queue code to its own module (finally)
– restructured queue interface to use rsRetVal and instances, removed
dependency on globals – now more like a real class
– implemented queue type “drivers”
– queue is now a full object and handles threading by itself
– applied Michael Biebl’s patch to clean up the makefiles
– added capability to use a linked list for queuing to the queue class
– added $MainMsgQueueType config parameter
– some cleanup
– added $SpoolDirectory config parameter
– added $MainMsgQueueFilePrefix config parameter
– begun working on disk queueing (not completed, do not use this mode!)
– begun some work on Msg Object serializiation
2008-01-04
– created a kind of general base class
– removed serialization pointer from queue; used new base class instead
– utilized the new auto-destruction capability so that the queue can now
destruct user objects if needed
– changed queue object Construction/Startup interface
2007-08-05
– added capability for concurrent access to the msg class. Can be dynamically
activated. If active, locking is employed.
– added the “direct” queueing mode to queue class (no queing at all)
– added multiple worker thread capability to queue class
– implemented $MainMsgQueueWorkerThreads config directive
– removed some no-longer-needed code (thanks Michael Biebl for the help)

A short rsyslog work log form the past days:

2007-12-31
– created omtesting, a debug and development aid output module. This is stage
work for the new queueing engine – we need a way to delay rule execution
and that’s what the module currently does ;)
2008-01-02
– released 2.0.0

Hi folks, probably the last rsyslog work log post for 2007. Thanks for sticking around – and hopefully I’ll see you again in 2008. It’ll become a very exciting year, with a lot of new features. I am eager to implement what is right now on my head, and I’ll most probably will start with modifying the message queue, an endeavor that will ultimately lead to store-and-forward capability just like in syslog-ng’s premium edition. And the good news is that I hope to finish that in January 2008 ;) — what also means that I have made up my priorities. Was not an easy job, and I hope I got it right. So store-and-forward with enhanced output threading is first and the other things will follow later. To me, the hardest decision was to put off expressions, another feature at least I would like to see the sooner the better.

But now back to the work log:
2007-12-27
– added $UDPServerAddress config directive
– added capability to have multiple UDP listeners running concurrently
– applied cross-platform patch from darix to facilitate GSS-API compile
on more platforms
– some cleanup
– internal restructuring in omfwd.c – stage work for further modularization
I think I also fixed a bug as a side-effect – but not looked to much at it
2007-12-28
– took TCPSend() apart and made it generic via function pointers
– moved TCPSend() and frame building code to tcpsyslog.c
– omgssapi created
– removed gss-api code from omfwd.c

I initially sent this message only to the mailing list. But now I think it make sense to reproduce it here. So there we go:

I am working on the modular structure of rsyslog v3. I am currently revisiting gss-api support. I notice that with the current omfwd, it will be extremely hard to separate gss-api support into its own module. Doing so will break backward compatibility to the configuration file.

GSS-API has been out only for a few days, and mostly over the holiday period. So it is much less of a concern if we introduce now some changes that will case rsyslog.conf format modifications. Much less trouble than when we release v2, a release expected to be in wide use for at least half a year, if not much longer. V2 released with the current syntax would require me to do some tricks in v3 to keep compatibility. Quite complex.

So I decided to create a omgssapi for v3 and extract the gss-api code from omfwd. It looks like this can be done without too much code duplication. There will be some duplicate code, but it will shrink as v3 continues to be developed. Once I have a good working version, which I expect very soon, I will backport that to the v1/2 source tree. I’ll then do a new v1 release with a slightly incompatible gss-api config file syntax. After this is out for a few days, I hope I can than finally push out that version as v2.

I hope this is a good decision. I think it will save us major future trouble at the expense of a relatively slight disturbance in the late v1 timeline. I guess most user’s won’t even notice there is a change.

As always, Feedback is appreciated.

I have made good progress with rsyslog‘s input modules. As it looks, the basic things are done and the input module interface has been proven to be both quite stable as well as very simple. It doesn’t yet support different instances, but I begin to think that I do not even need them – also not in the long term.

Of course, most of the current input modules are not clean modules. They have a lot of dependencies to other parts of the code, which is not yet able to be dynamically loaded. But at least there is a foundation on which additional modules could be build. Getting the current input modules to be real clean modules will require further stage work. Many thanks need to be done.

So what to do next? It now comes down to both a matter of priorities and dependencies. I am writing this note here mostly for myself. It helps my clear up my thoughts and will also probably serve as a reference for quite a while. My thoughts may be hard to understand – sorry for that. But I thought I make them public when I write them down – even if they are not really targeted toward others. I still hope they may help you get some more background info.

So what’s to do:

- find a way to handle global settings
- multi-threaded output modules
  a perquisite for
  - create queued outputs (write to queue if action fails and 
    restart when it resumed)
- re-write way config file is read
  probably perquisite for:
  - create expression support
    - in templates
    - in selector filters
- create interface for (loadable) user function modules
- create a system to allow loading "library" loadable modules
  (e.g. network library for imudp, imtcp, ...)
  - separate GSSAPI from plain TCP (requires libs and lib extension system)

I’ve been a bit busy with rsyslog over the xmas break. Here are the changes:

2007-12-25
– moved some more net functionality out of syslogd.c – stage work
– fixed duplicate license text in syslogd.c – made it ambigious
– moved udp net code – again, stage work
– moved some of the udp input code to its right place
2007-12-26
– moved cross-platform define for AI_NUMERICSERV to net.h
– made udp code somewhat less dependable on global variables — stage work
– removed omfwd code dependency on “finet”
– removed imudp code dependency on “finet”
– removed active INET code from syslogd.c – still some auxiliary things
remain
– fixed socket leak in omfwd.c
– removed global variable LogPort
– removed gloable variable AcceptRemote and external def of bFinished

It has been a good day today! Finally, the alarm() call has been deleted! :) That was something long on my agenda, but I couldn’t do it without the redesign of the inputs. The alarm() was not really a big issue, but it became an annoyance to me because it was so hard to remove.

I would also like to mention that I will do only occasional work during the holiday period. So I do not expect more serious changes until early January. Some releases, however, are due next week (maybe 2.0.0).

Here is the detailed rsyslog worklog for today:

– removed no longer needed mutex from omfwd
– released a preview of 3.0.0 “as is” to mailing list – just to get the idea
– begun work on imtcp
– created first version of imtcp (still very much depending on syslogd.c for
configuration and a lot of other things)
– cleaned up code (resulting in some shuffeling from syslogd.c to the
“right” module)
– prepared for imudp
– created an initial version of imudp.c. The majority of UDP reception code
is now in that module and it is dynamically loadable. HOWEVER, that doesn’t
mean it is a proper module. There are still many, many dependencies on
global variables, cross-module calls and such. However, havin the code base
separated allows me to carry out some other cleanup before I return to
create a really clean implementation of these modules. So it is kind of a
stage work. Just don’t mistake it with “the real thing”…
– removed code no longer needed
– finally, alarm() has gone away :) — this is now done by the main thread
– some cleanup

Yesterday was a very busy day for rsyslog. I am on a good path to input modularization, but the hardest part needs still be done ;)

Here is the log:

– bugfix: fixing memory leak when message queue is full and during
parsing. Thanks to varmojfekoj for the patch.
– working on a potential race condition on the new input module
interface. See newsgroup posting for details on the issue:
http://groups.google.com/group/comp.programming.threads/msg/330b9675f17a1ad6
I tried some mutex operations but came to the conclusion that this
does not really help. So I have now switched to plain thread
cancellation, which so far seems to be OK. Need more practical
experience with other input modules to make a final decision. Thus
I leave all code in and have just disabled the problematic
code.
– implemented $klogUseSyscallInterface config directive
– implemented $klogSymbolLookup config directive
– moved unix socket code to its own module (imuxsock)
– implemented $OmitLocalLogging config directive
– bugfix: memory leak in cfsysline.c/doGetWord() fixed
– implemented $SystemLogSocketName config directive
– implemented $AddUnixListenSocket config directive
– MILESTONE reached: imuxsock initial version done
– removed single-threading support for sending TCP messages; caused
simplyfication of output module interface as well as core syslog
processing.
– moved udp send code to its own function

Here is my recent rsyslog work log:

2007-12-18
– removed files from cvs that not belong there (thanks to Michael Biebl for
pointing that out)
– restructured #include’s somewhat thanks to Michael Biebl
– code cleanups thanks to Michael Biebl
– applied Michael Biebl’s patch to enhance $includeconfig to support
wildcard filenames
2007-12-19
– applied some more cleanup provided by Michael Biebl
– applied enhanced gss-api functionality provided by varmojfekoj
– GSS-API support for syslog/TCP connections was added. Thanks to
varmojfekoj for providing the patch with this functionality
– release 1.21.0
– added the -c option
– enhanced -c option support (some basics)
– bugfix: llDestroy() left the list with invalid root/last pointers