Yesterday’s rsyslog work log:
2008-03-06
– applied patch from Michael Biebl to clean up build system
– bugfix: gssutil was not linked against gss libs (bug introduced
yesterday)
– fixed a few remaining logerror() calls – thanks to Michael
Biebl for pointing that out
– changed –enable-klogd to –enable-klog as suggested by Michael Biebl
– bugfix (yesterday’s bug): abort during hup if library module was loaded
– portability: no longer using pthread mutex auto-init macros as they
seem to be buggy on some platforms
– worked some more on HP UX – got it to compile and receive UDP messages
– added capability to ignore client-provided timestamp on unix sockets and
made this mode the default; this was needed, as some programs (e.g. sshd)
log with inconsistent timezone information, what messes up the local
logs (which by default don’t even contain time zone information). This
seems to be consistent with what sysklogd did for the past four years.
Alternate behaviour may be desirable if gateway-like processes send
messages via the local log slot – in this case, it can be enabled
via the $InputUnixListenSocketIgnoreMsgTimestamp and
$SystemLogSocketIgnoreMsgTimestamp config directives
– applied some bugfixes to new code supplied by varmojfekoj – thanks!
– released 3.12.1
rsyslog work log 25
Yesterday’s rsyslog work log:
2008-03-05
– updated omgssapi to use the current interface version definition
(I seem to have forgotten this yesterday)
– changed modules.c calling conventions to be interface-based
– moved module loader from conf.c to module.c, where it belongs
– made the necessary plumbing to auto-load library modules
– upgraded debug system to include iRet in function exit message
– changed module interface so that instances need only to be
supported by output plugins (if we actually need them for input
plugins, we can always add it again…)
– milestone: first implementation of library modules (but do not
get unloaded on exit/hup so far)
– bugfix: imfile used invalid obj-call (newly introduced bug)
– extracted logerror*() family of functions from syslogd,
made them their own class and converted to new object calling
conventions (interface-based)
– converted gss-misc into a loadable library module
– bugfix: actions were not correctly retried; caused message loss
– moved date/time handling functions to their own object
– changed module loader to automatically add “.so” suffix if not
specified (over time, this shall also ease portability of config
files)
– shuffled some more code from syslogd.c to the right places
– fixed bug in build process
rsyslog work log 27
Yesterday’s rsyslog work log:
2008-03-03
– made imgssapi compile
– fixed newly introduced bugs in imgssapi and imtcp and their helpers
now plain tcp works again
– tried to setup a compile environment on Opensolaris and HP UX. That
was really a frustrating experience, they make it very hard to get
up and running. I didn’t manage to come to a point were I could compile
rsyslog, so I can’t find any portability issues. However, that’s not
high on my priority list, so until I receive further interest, I’ll
not further pursue anything in that direction. And interest means that
somebody should be around with advise on how to setup a compile
environment. I’ll gladly work from there on, but I don’t bother
spending precious time just to get all this boxes up and running…
rsyslog work log 28
Past day’s rsyslog work log:
2008-02-29
– corrected page title in omsnmp doc
– some stage work on rsyslog loader
– changed obj base object’s calling interface to use the new obj_if_t
interface structure
– tweaked omgssapi a bit to use our regular calling conventions
– added “debug” command to debug environment settings
– changed the object/interface system to use a new way of calling, giving
up the numerical object ID. This was necessary as we needed more
extensibility for third-party modules (which don’t play at all with
the previous fixed object ID). This is stage work for the object loader.
Please note that I needed to change the object (de)serializer, I can’t
outrule that I have introduced bugs there.
– corrected invalid function definition
– converted conf.c to an abstract class
2008-03-01
– added class tcps
– added class tcps_sess
2008-03-02
– changed imtcp to use new classes; seems to work; imgssapi currently broken
rsyslog work log 29
Yesterday’s rsyslog work log:
2008-02-28
– updated “mysql paper” to include information for other databases, too
– worked a bit on the rsyslog/syslog-ng comparsion – slowly gets in better
shape ;)
– wrote doc on how to use the expression engine
– changed ABNF to fully support old property names
– added case-insensitive comparison operations
– MILESTONE: initial expression support completed
– (finally;)) released 3.12.0
– implemented environment-settable debug options
– enabled debug-support to pull runtime options from environment (bug 18)
– bugfix: removed debugging code that I forgot to remove before releasing
3.12.0 (does not cause harm and happened only during startup)
– added “help” command to runtime debug flags
– switched to git as a test run; currently I maintain both the cvs
and git repositories (via manual file copies). If all goes well, will switch
to git-only when I feel good about it (a week or so?)
rsyslog work log 30
Yesterday’s rsyslog work log:
2008-02-27
– bugfix: queue cancel cleanup handler could be called with
invalid pointer if dequeue failed
– bugfix: imfile could abort under extreme stress conditions
(when it was terminated before it could open all of its
to be monitored files)
– bugfix: queue disk file were not properly persisted when
immediately after closing an output file rsyslog was stopped
or huped (the new output file open must NOT have happend at
that point) – this lead to a sparse and invalid queue file
which could cause several problems to the engine (unpredictable
results). This situation should have happened only in very
rare cases. tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=40
– bugfix: during queue shutdown, an assert invalidly triggered when
the primary queue’s DA worker was terminated while the DA queue’s
regular worker was still executing. This could result in a segfault
during shutdown.
tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=41
– bugfix: object property deserializer did not handle negative numbers
– bugfix: queue aborted when it was shut down, DA-enabled, DA mode
was just initiated but not fully initialized (a race condition)
– bugfix: queue properties sizeOnDisk, bytesRead were persisted to
disk with wrong data type (long instead of int64) – could cause
problems on 32 bit machines
– fixed a problem introduced today, on-disk queue size was now
wrongly calculated (but not in any released version)
rsyslog work log 31
Yesterday’s rsyslog work log:
2008-02-26
– implemented data type conversion
– released 3.11.5
– implemented majority of comparison operations
– added some temporary testing aids to conf.c, so that we can debug
expression support as it is implemented
– fixed a couple of bugs in expression system
– added more operations to virtual machine – now works well with
constants
– added PUSHMSGVAR operation to vm
– included expression support in filter module (and it works ;))
– added sysvar class
– added PUSHSYSVAR operation to vm
– added string concatenation operator & to RainerScript
– fixed segfault when pure string values were tried to be added
– some stage work for library modules
– fixed wrong pointer in imgssapi config coding
– added $InputGSSServerMaxSessions config directive
– worked on queue stability
Why RainerScript?
After my initial announcement of RainerScript, I got some good feedback. I’d like to reproduce an especially useful discussion here:
> I wondered if using a complete scripting language for configuraton
> is not making it overly complex and error prone. I specifically means
> function definition etc. here.
The key is that I will model it so that the complexity is only seen when
it is actually needed. Most importantly, it will continue to understand
plain old syslog.conf format. So most peoply will probably never see the
scripting features. HOWEVER, I think it is useful to have these for a
few select folks who really need them. Function definitions, for
example, will counter syslog-ng’s feature to define a filter. I don’t
know if it is actually worth and useful to define a filter to reuse it.
But with the function, you can do it. So those coming from syslog-ng can
find their favorite feature ;)
I think it is probably worth to think about how I came to the idea of
scripting support. The root cause is complex logic, complex expressions.
A few days ago I started to implement them … and quickly found out
that actually the easiest way is “to do it right”. So I now have a
parse, bytecode and an interpreter/virtual machine running in rsyslog.
All of this “just” to support complex expressions. While I thought about
it, I finally realized that it takes just a little bit to extend these
things so that the end result will be an easy to use scripting facility.
That also solves the config file issues that’s dangeling so long and, as
a side-effect, brings up some interesting options (for those that have
need for it).
For example, I thought about how to best handle rewriting message
content. Now, the solution is obvious. I’ll implement a “set” facility
in the language that enables message variables to be rewritten on the
fly (e.g. for folks who would like to drop some parts of the message). I
agree, all not pretty mainstream, but useful in some cases. And keep in
mind that it comes at a (complexity) cost only if you need it. That’s
based on the fact that expression support was originally planned as a
by-line of the current config format, so it needed to work with the rest
in any way ;)
> But if you really need that kind of flexibility and scripting
> capabilites, have you considered to use existing languages, like lua
> , which were specifically designed to be embedded into
> applications?
Not really, because of the way it has evolved. A know a new language is
already evil, but I see big benefit for the project this time. The
reason is that I can very tightly integrate it into rsyslog’s features
like the upcoming loader. So it will be very cleanly and efficiently
integrated. For example, I don’t think that I could support old-style
format *just within* the new style format with any third-party engine.
rsyslog work log 32
Yesterday’s rsyslog work log:
2008-02-25
– implemented data type conversion
– released 3.11.5
– implemented majority of comparison operations
– added some temporary testing aids to conf.c, so that we can debug
expression support as it is implemented
– fixed a couple of bugs in expression system
– added more operations to virtual machine – now works well with
constants
– added PUSHMSGVAR operation to vm
– included expression support in filter module (and it works ;))
– added sysvar class
– added PUSHSYSVAR operation to vm
– added string concatenation operator & to RainerScript
– fixed segfault when pure string values were tried to be added
rsyslog work log 33
Past day’s rsyslog work log:
2008-02-22
– integrated patch from Peter Vrabec to change the build process to produce
imtcp.so and imgssapi.so from imtcp.c (in support of new gassapi
input module) – many thanks, Peter!
– applied patch by varmojfekoj to allow gssapi functionality to be
build as a separate plugin (so that gssapi and plain tcp
functionality can be individually distributed). Also inclulded
some other enhancements, most importantly initial compatibility
mode system
– added some information on old-style config to ABNF (probably
useful in getting us to a new config format)
– fixed bug in duplicate module load detection
– corrected config doc – IETF no longer intends to add facilities
– updated compatibility notes doc
– added some doc for imgssapi and imtcp input modules
– completed initial vmstk implementation
– begun implementing rsyslog virtual machine (vm class)
– worked a bit on type conversion (specified the interface)
– simplified var object, now only supports strings and numbers as
a single type
– worked a bit on var_t data type conversion
2008-02-24
– added some thoughts on RainerScript
– worked a bit on conversion functions