Author: Rainer Gerhards

OK, I am guilty of lazyness… I worked quite a bit on RELP and by doing so, I always thought “it’s not much for the rsyslog work log, so let’s do it tomorrow” ;) Well, tomorrow is now two weeks old and I think it is a good plan to come back to my habit of posting work logs. This time, of course, a bit more than usual.

Please note that the initial version with relp support, 3.15.0 has been released. I am currently evaluating the next focus feature, which probably is native TLS support. If so, I’ll probably again need to put some work into a utility library, which will not directly be rsyslog related. But I hope I’ll still continue to post something useful here. In the mean time, please be reminded that any implementation reports are still most welcome.

So here we go, the past ca. two-weeks worth of rsyslog work log:
2008-03-19
– added flow control options to other input sources
– worked on librelp
2008-03-20
– bugfix: some minor memory leaks
– bugfix: some slightly invalid memory accesses
– made librelp and rsyslog relp system send the first messages
to the remote peer (but it then discards them ;))
– made debug module free some memory on exit to make memory debugger
happy
2008-03-21
– added capability to receive RELP messages and forward them to the
main message queue to imrelp (not yet fully finished)
– cleanup of omrelp
– prepared omrelp for real “relp action”
2008-03-22
– changed queue’s discard severities default value to 8 (do not discard)
to prevent unintentional message loss
– removed a now-longer needed callback from the output module
interface. Results in reducing code complexity.
– Greatly enhanced rsyslogd’s file write performance by disabling
file syncing capability of output modules by default. This
feature is usually not required, not useful and an extreme performance
hit (both to rsyslogd as well as the system at large). Unfortunately,
most users enable it by default, because it was most intuitive to enable
it in plain old sysklogd syslog.conf format. There is now a new config
setting which must be enabled in order to support syncing. By default it
is off. So even if the old-format config lines request syncing, it is
not done unless explicitely enabled. I am sure this is a very useful
change and not a risk at all. I need to think if I undo it under
compatibility mode, but currently this does not happen (I fear a lot of
lazy users will run rsyslogd in compatibility mode, again bringing up
this performance problem…).
– added $ActionfileEnableSync config directive
– bugfix: internally generated messages had “FROMHOST” property not set
2008-03-25
– bugfix: continue parsing if tag is oversize (discard oversize part) – thanks
to mclaughlin77@gmail.com for the patch
– added $HHOUR and $QHOUR system properties – can be used for half- and
quarter-hour logfile rotation
– bugfix: QHOUR and HHOUR properties were wrongly calculated
2008-03-27
– made relp modules use new relpengine-provided feature selection functions
– bugfix: fixed memory leaks in stream class and imfile
– bugfix: $ModDir did invalid bounds checking, potential overlow in
dbgprintf() – thanks to varmojfekoj for the patch
– changed default for “last message repeated n times”, which is now
off by default
– implemented backward compatibility commandline option parsing
– bugfix: -t and -g legacy options max number of sessions had a wrong
and much too high value
– automatically generated compatibility config lines are now also
logged so that a user can diagnose problems with them
– added compatibility mode for -a, -o and -p options
– MILESTONE: compatibility mode processing finished
2008-03-28
– updated man pages
– changed default file output format to include high-precision timestamps
– added a buid-in template for previous syslogd file format
– added new $ActionFileDefaultTemplate directive
– added support for high-precision timestamps when receiving legacy
syslog messages
– added new $ActionForwardDefaultTemplate directive
– added new $ActionGSSForwardDefaultTemplate directive
– added build-in templates
– fixed small memory leak in tcpclt.c
– bugfix: fixed small memory leak in template regular expressions
– bugfix: regular expressions inside property replacer did not work
properly
2008-03-29
– removed –enable-mudflap, added –enable-valgrind ./configure setting
2008-03-31
– bugfix: tcp receiver could segfault due to uninitialized variable
– docfix: queue doc had a wrong directive name that prevented max worker
threads to be correctly set
– worked a bit on atomic memory operations to support problem-free
threading
– added a –enable/disable-rsyslogd configure option so that
source-based packaging systems can build plugins without the need
to compile rsyslogd
– released 3.13.0-dev0
– begun 3.17.0 [focus: TLS for plain tcp syslog]
2008-04-01
– bugfix: rsyslogd was no longer build by default; man pages are
only installed if corresponding option is selected. Thanks to
Michael Biebl for pointing these problems out.

Today’s rsyslog work log:
2008-02-01
– applied documentation fix by Michael Biebl — many thanks!
– added input-plugin interface specification in form of a (copy) template
input module
– cleaned up some no longer needed files, thanks to Michael Biebl for pointing
this out
– added sample config file to distribution tarball
– bugfix: immark did not have MARK flags set…
– very quickly hacked a rought outline of the file monitor (without any
guarantees)

Yesterday’s rsyslog work log:
2008-01-31
– added some advanced features usually in demand to sample rsyslog.conf,
but left it commented out
– rename $TimoutWorkerThreadShutdown to $WorkerTimoutThreadShutdown
for consistency reasons
– changed default for action queue size to 1000 elements (more reasonable here)
– fixed bug in sample rsyslog.conf
– fixed wrong action suspend/resume handling
– we have some issue with the mutx in dbgoprint, but that is acceptable for
the time being, I just removed the deadlock codition (debug system only)
– bugfix: dbgoprint mutex – was too simple once I wrote the tracker item ;)
– bugfix: having fun with 32/64 bit portability – after 15 years, I finally
was trapped again ;) — now fixed, sizes > 2GB supported on 32bit platforms

Yesterday’s rsyslog work log:
2008-01-30
– renamed Msg object to usual all-lowercase object name (else we ran into
troubles with the framework, also it was somewhat ugly…)
– fixed a memory leak in object destruction (was recently introduced by
object naming, not present in any released version)
– finally implemented infinite action retries via -1 retry count
– fixed a bug that caused $MainMsgQueueCheckpointInterval to work incorrectly
– fixed bug in output module interface, see
http://sourceforge.net/tracker/index.php?func=detail&aid=1881008&group_id=123448&atid=696552
– changed the ommysql output plugin so that the (lengthy) connection
initialization now takes place in message processing. This works much
better with the new queued action mode (fast startup)
– fixed a newly introduced bug that caused output module’s doAction entry
point to be called on more than one thread under some circumstances
– fixed a bug that could cause invalid string handling via strerror_r
varmojfekoj provided the patch – many thanks!
– adopted varmojfekoj’s patch to v3, some more sterror_r’s ;)
– bugfix: MsgDup() did not work with new base object data structure
– implemented simple output rate limiting
– addded $ActionQueueDequeueSlowdown config directive
– addded $MainMsgQueueDequeueSlowdown config directive
– implemented limiting disk space allocated to queues
– addded $MainMsgQueueMaxDiskSpace config directive
– addded $ActionQueueMaxDiskSpace config directive

Yesterday’s rsyslog work log:
2008-01-29
– improved debug support a bit (assertions)
– restructured code, moved some part out of syslogd.c to action.c, where
they belong (still some more to do in that regard ;))
– moved correct retry logic into action processing queue
– removed debugging support from sync class, debug class now provides
much more
– implemented naming for all objects (mostly as a debug aid, but you never
know what else it will be good for)
– converted queue.c to use dbgoprint() instead of dbgprintf()
– fine tuning on queue naming
– action queues are now also named (otherwise you can’t read the debug log ;))
– bugfix: added forgotten docs to package
– bugfixing newly added action code
– action queue params are reset to default for each action
– added ability to re-enqueue objects into the queue when a worker thread is
cancelled

Yesterday’s rsyslog work log:
2008-01-28
– added capability to record last known exec location in debug stack dump
– added some doc on how to handle massive amounts of database inserts
with rsyslog
– some more testing and cleanup with the queue class (pretty stable now)
– cleanup to prepare for release
– PID file is now also written in debug mode
– fixed a segfault issue with the rsyslog shutdown exit debug message when
in release build
– added “runtime instrumentation” feature to ./configure
– some more cleanup
– add Michael Biebl’s autoconf patch for netsnmp – thks!
– released 3.10.3 – finally ;)
– implemented the $ActionResumeRetryCount config directive
– added queue between main queue and action executor (currently
works in “direct” mode only, else crashes)
– added $ActionQueueFilename config directive
– added $ActionQueueSize config directive
– added $ActionQueueHighWaterMark config directive
– added $ActionQueueLowWaterMark config directive
– added $ActionQueueDiscardMark config directive
– added $ActionQueueDiscardSeverity config directive
– added $ActionQueueCheckpointInterval config directive
– added $ActionQueueType config directive
– added $ActionQueueWorkerThreads config directive
– added $ActionQueueTimeoutshutdown config directive
– added $ActionQueueTimeoutActionCompletion config directive
– added $ActionQueueTimeoutenQueue config directive
– added $ActionQueueTimeoutworkerThreadShutdown config directive
– added $ActionQueueWorkerThreadMinimumMessages config directive
– added $ActionQueueMaxFileSize config directive
– added $ActionQueueSaveonShutdown config directive

Past two day’s rsyslog work log:
2008-01-26
– improved SIGSEGV handler a bit (now we get a core dump)
– partially fixed bug that caused rsyslogd to stall processing enqueued messages
after turning off DA mode and before any new message were arrived (if a new
message arrived, everything went back to normal, so it was a temporary halt)
2008-01-27
– fixed the situation where message processing could be stalled for some period
after DA mode turn off
– reduced number of unnecessary wakeups of DA worker thread when high water mark
is not yet reached
– fixed a cosmetic issue in timeoutValue calculation (just used for displaying)
– fixed a bug when shutting down DA queue
– some more fixing and cleanup on the queue shutdown sequence
– fixed queue termination in case bSaveOnShutdown is 0
– implemented the $MainMsgQueueSaveOnShutdown config directive
– implemented the $MainMsgQueueWorkerThreadMinimumMessages config directive
– implemented the $MainMsgQueueTimeoutWorkerThreadShutdown config directive

Yesterday’s rsyslog work log:
2008-01-25
– restructured queue shutdown so that the queue timeout is properly applied before
terminatiing the queue
– disk-assisted queue mode finally begins to look good ;)
– added debug-support environment variable RSYSLOG_DEBUGLOG
– fixed worker timeout which was accidently not set (and thus 0)

Long time no rsyslog work log post. But this time its a rather long one:

2008-01-14
– removed no longer needed “Initialized” logic and variable
– worker shutdown sequence enhanced to try different ways to shut
down and terminate workers if none helps (this protects against
badly written output plugins which hold the queue for too long)
– implemented $MainMsgQueueTimeoutActionCompletion config directive
– implemented $MainMsgQueueTimeoutEnqueue config directive
– implemented $MainMsgQueueTimeoutShutdown config directive
– some cleanup
– removed $MainMsgQueueImmediateShutdown config directive and handling,
this has been superseeded by the Timeout params. Backward compatibility
is no concern, no version with that directive was ever released.
– released 3.10.2
– fixed a bug with standard template definitions – thanks to
varmojfekoj for spotting it
– implemented config file handlers for
* $MainMsgQueueHighWaterMark
* $MainMsgQueueLowWaterMark
* $MainMsgQueueDiscardMark
* $MainMsgQueueDiscardSeverity
but did NOT yet implement the functionality behind these directives!
– implemented $MainMsgQueueDiscardMark and $MainMsgQueueDiscardSeverity
(but serverity needs to be specified numerically for the time being)
– also implemented $MainMsgQueueDiscardMark logic on dequeue, changed
defaults
– implemented $MainMsgQueueDiscardMark == 0 –> disable Discard logic
– implemented $MainMsgQueueSize == 0 –> no limit on queue size
– begin disk assisted queue memory queue modes (not fully implemented
yet)
2008-01-15
– disk assisted queue works quite well, except for startup from disk queue
– changed startup of disk assisted mode to allow for higher concurrency,
most importantly allow the input to continue enqueue msgs while the
disk queue is initialized. This may help somewhat with UDP and other
lossy sources
– improved shutdown processing – in-memory queue is now drained to disk
– first shot at queue restore on startup, but could not finish before
I need to leave ;)
2008-01-16
– fixed a compile-time bug in release mode spotted by Michael Biebl
– queue is now able to restore persisted state on startup (but still some
fine tuning to be done)
– re-released 3.10.2 due to bug reported by Michael Biebl
– cleaned up queue disk startup
– implemented dynamic startup and shutdown of worker threads based on
current activity
– some cleanup and fixes
– some more cleanup and flagged places where we need to implement
DA-input-only mode
2008-01-17
– changed obj_t destructor interface
– worked on threading
– fixed sync issue on shutdown process if need to persist pure memory
queue to disk
– adapted DA logic to the ability to shut down all workers for inactivity
2008-01-18
– created an in-depth description of DA assisted queue mode
– snapshot of new thread coding – DA mode still does not work, but need
to save
2008-01-19
– seperated mutex for queue size management from those for queue thread
management
– some further cleanup on the mutexes
2008-01-20
– begun object model redesign for clearer design of queue class, causes
creation of wti and wtp classes
– initial creation of wti class (worker implementation missing)
2008-01-21
– continued implementing wti class
– added some mutex-support to srUtils
– initial creation of wti class (still under development)
2008-01-22
– continued to work on queue
– implemented debug interface
2008-01-23
– added mutex debug instrumentation
2008-01-24
– improved debugging support (faster, less memory used, function invocation
count is maintened)
– redesigned queue to utilize helper classes for threading support. This is finally
in a running state for regular (non disk-assisted) queues, with a minor nit
at shutdown. So I can finally commit the work again to CVS…

I’ve not posted anything the past days – not because I don’t do anything but because I am terrible busy with new features. The new massively multithreading engine is giving me a somewhat harder time than I expected (and I have to admit a design flaw which I am recovering from). Everything else is currently being pushed away by my desire to finish store-and-forward enabled actions by the end of the month. I am no longer sure I’ll meet that deadline, but at least I’ll try.

Updates to the blog will hopefully resume soon, as will new releases. I’ll post work logs soon.

So you now know at least what’s going on… BTW: I am looking for some test environments where store-and-forward and/or high performance (running on as many cores as possible) is in high demand.