I thought quite a while about the upcoming configuration file format today. I
have to admit, I am not very satisfied with the outcome. I have a few potential
formats, but either they are too brief (and quite complex) or there is a lot of
text. Also, I am not satisfied yet with the flexibility of the format. I guess
this is a harder nut to crack than I thought. Unfortunately, I can not yet post
any of the ideas, as I have done them on paper and they are probably unreadable
to anybody but me. Bear with me, I’ll create an online version these days.

But to understand any config file formats, you need to know more about the
object model. A first version of that model has materialized on my mind, and
actually rsyslog 1.18.0 and above is already supporting parts of it. To aid
understanding (and get some discussion going), I have created a quick outline of
what I have in mind. It’s textual, it’s incomplete and it can not be formally
verified. Still, I think (sincerely hope), it is useful. So I decided to post
what I currently have. Besides the actual object model discussion, it will help
a great deal in understanding what needs to be in the config file (thus it’s
currently a bit biased towards configurable objects and internal ones are almost
completely missing).

rsyslog v3 Objects

Rsyslog uses objects, even though it is written in C. This can quite well be
done. Only at some points (like inheritance) we need to fiddle a bit with the
language. But nothing to hard to be done. We try to keep object overhead very
low, so it should be more like traditional C than C++.

module

The module object is the base class for anything that even remotely looks
like a loadable object. In the long term, that might eventually be almost
everything. It handles the necessary plumbing like loading modules, keeping
track of module status, querying interface and all those things… It does not
by itself provide any module-specific actions.

input

An input gatheres messages (events) from external sources. Current typical
examples are UDP or TCP based syslog. However, there is no architectural limit,
so in the long term an input module may also gather SNMP traps or file lines.
Please note that an input does not necessarily parse the obtained event by
itself – this may be delegated to a parser module (this whole thing still needs
to be thought out).

output

An output module receives strings from the engine and writes them to some
ultimate destination. Popluar examples are files, databases or remote syslog
servers.

action

The action object is the "engine wrapper" around an output module. It
provides numerous servies to the output. Most importantly, it provides the core
plumbing behind restarting and queueing actions.

function

A function object (and lodable module) provides extensibility for internal
processing. Version 3.0 will support programming-like functions, which replace
and extend the property replacer options. For example, we may want to extract
characters 5 to 10 and convert them to lower case. With current rsyslog, this
works as follows: %msg:5:10:lower%. With Version 3, it will look much nicer (at
least I hope so): lower(substr(%msg, 5, 10)). The exact sytax and semantics of
how this is used in the config file is under development – but I think you can
get the idea from the example. The core improvement featuere-wise is that with
functions, rsyslog can very esaily be extended by just programming a small
plug-in with a new function. Do to the programming-like strucuture, the new
function can be combined with all exisitng ones.

Functions will be supported everywhere a string is supported, which means
everywhere. Examples are filters, output format (templates) and file name
generation.

Message Flow

This is a brief description of how a message (an event) flows inside rsyslog.

The message text is received by the input module. The input module, possibly
with the help of a parser module uses the message text to create a msg-in memory
structure. This msg structure enters rsyslog’s main processing queue. As soon as
possible, the msg object is dequeued. Now the rule engine processes the rule set.
A rule set object contains multiple rule object, which are executed in order
until either all rules are processed or a discard action is encountered. If
there are multiple rule set objects bound to a single input module, they are
also executed in the configured order (the normal discard action will discard a
message for one rule set, but an additional rule set will  continue to
process the msg object. A special case of the discard action can be envisioned,
which discards a msg object for all bound rule sets.. Each rule object contains
filter objects and action objects. First, all filter objects are executed. That
means, their expressions are evaluated. If there are multiple filter objects,
their result is combined in a AND operation (other modes are not supported –
that could be easily done by crafting a specific filter expression, so we do not
encourage an additional set of complexity be allowing additional boolean
operators when multiple filter objects are used inside a single rule). If the
outcome of the overall filter object evaluation results in true ("to be
processed"), all action objects are called. The are called in the order of their
appearance in the config file (no exceptions here). The msg object is used to
generate that string required by the action (note that the msg object itself is
NOT passed to the action, this is a security and encapsulation boundary). For
string generation, function plug-ins may posibly be called (this is also the
case during filter procesing). The resulting strings are  passed to the
execute method of the action object. That object decides if queing,
rate-limiting or other functionality generically available to all actions is the
be carried out. All of this is done by the action object itself. Finally, the
strings are passed from the action object to the actual output object (a plug-in).
The output objct takes the strings and performs whatever processing needs to be
done. After all actions, rules and rule sets have been processed (or when a
discard action occurs), the msg object is destroyed. Please note that even than
a copy may be held in memory, because that might be needed for duplicate message
reduction and similiar features. Thus, msg is reference-counted and actually
only destroyed when the count reaches zero.

Please note that there are a number of utility objects involved. For clarify,
these have been omitted. Also, all function calls return a meaningful return
state. The caller will process that state with care. In some cases, however,
this may mean ignore a failed call, because this is the most appropriate thing
to do (e.g. when an action fails – if we’d abort processing the msg object, we’d
do much more harm).

Data flow

This is a brief and not so accurate picture of the data flow. It concentrates
on the role of plug-ins.

input module(parser) -> rule engine (custom functions) -> action (with
aggregated output module)

Known Open Issues with Plug-ins

The object model is not yet fully designed, so there is nothing bad about
open issues. I list them, so that the do not get lost.

Probably the number one question is how message-modifying plug-ins can be
created. I think about things like implementing syslog-sign or TLS security. It
is my current, unverified, view that we must have something like "filter-plug-ins"
which can be placed inside the message data flow to modify the message text and/or
the msg object.

Copyright © 2007 by
Rainer Gerhards

Last Updated: 2007-08-30

Rsyslog currently offers only very rough support for plug-ins: only the MySQL plugin is available as a plugin. Even the output plug-in structure is not yet finished. So it looks like plug-is are quite unimportant to rsyslog. Right?

Wrong! Totally wrong! If there is a single feature that is absolutely critical to rsyslog, it is plug-in support. Plug-ins offer a way to provide dynamically loadable extensions of the core functionality. Provided the plug-in interface (API) is clean and well-designed, that can come without any performance hit.

Now don’t just think about plug-ins for exotic features. What if almost everything is a plug-in? Well, in that case the code loaded in memory (the footprint) can be exacty tailored to what is needed. This is most benefitial for small, embedded system with limited resources available. On the other hand, this doesn’t limit a highly capable system of offering a very rich functionality. It just loads different plug-ins.

Remember that almost all functionality can be implemented as a plugin. Inputs (E.g. UDP syslog or local domain sockets) and outputs (e.g. database output or message forwarding) make somewhat natural plugins. But plugins can also be used in filtering. For example, the (expensive) regexp functionality could be provided by a plugin. As could be different message parsers.

This is exactly the way rsyslog is heading: we will try to provide an ultry-slim framework which offers just the basic things needed to orchestrate the plug-ins. Most of the functionality will indeed be available via plug-ins, dynamically loaded as needed.

I am not yet sure when we will reach his goal. But the object model design and the overall design direction heads into this area. With that design philosophy, we can make rsyslog really universally available, even on low-powered devices (loading just a few plug-ins). At the high end, systems with a lot of plug-ins loaded will be able to handle the most demanding tasks.

And, of course, plug-ins will hopefully not only be provided by the rsyslog project itself. It is our sincere hope that others will also begin to contribute functionality in the form of plug-ins. To facilitate that, we strive to create very simple plug-in interfaces. For example, all threading details will be handled by the core engine itself and plug-ins will be able to think they are running on a single thread. The core engine will handle the proper sync operations. Of course, plug-ins in need to use themselves a sophisticated threading model can do so – but it is not a requirement. Hopefully, that’ll make it very easy to create plug-ins and thus be very easy to contribute extended functionality for rsyslog. That, we hope, will help create a lively community and thus a breakthrough syslog project.

A strength of rsyslog is that it support stock syslogd’s config file format. One of its weaknesses, however, is that it support stock syslogd’d config file format.

Sound strange? Jepp – but it isn’t. Understanding the old config format is great when it comes to replacing stock syslogd. It’s easy for package maintainers and it is also easy for users (who do not necessarily need to learn anything new). However, that format is clumsy, especially when compared to syslog-ng. So there is already a lot of criticism for supporting that format.

To make matters worse, rsyslog’s enhanced functionality requires some enhanced configuration directives. They have evolved as rsyslog has evolved. Currently, they are only somewhat ugly. But the more the project evolves, the more ugly they will become.

In short words: I do not like the current config file format. But on the other hand, I also do not like spending much time “just” on creating another config file parser. And I do not like the learning curve associated with it.

Seth Vidal, some weeks ago, recommended to have a directive in the config file specifying whether it is old-style or new-style. I’ll probably follow this route. That would enable us to keep backwards compatible and allow users to re-use their knowledge. In “old format mode”, features would be limited by the file format (and the users’ ability to deal with it). The full glory would only be available in “new format mode”. That sounds just fair, especially as users need to learn the new functionality and then it does not really matter if the need to learn a new conf format while they go along.

OK, that seems to be (somewhat solved).

But what format should be the new style format? Seth Vidal recommended a natural language like format (see his post on the rsyslog mailing list). However, there are many things unanswered. We don’t just have filters, we have many things.

Let’s go to the root question: what is a config file good for? In my personal view, it should

• enable the user to specify what the product should do in an easy and simple to follow way
• allow the application to create internal control structures that specify how the application works

Both are equally important. But the second one tells us the at least part of the config file structure is dictated by the applications object design (or call it [mnodular] structure).

So I think to do the config file right, one first needs to know the application’s object model. For rsyslog, this is not yet published. Even further: it is not yet fully designed. The 1.x versions, as well as part of the 2.x version does not have a real object design (but 2.x has at least a partial one and is somewhat like an interim release on a way to the full model).

As a side-note, this is also the reason stock syslogd and early rsyslog works with such a simple config file: it is a monolithic application and such a one is served perfectly by an equally-monolithic config file. I intend to publish a full object model for rsyslog 3.0 in the fall/winter 2007 time frame. Then, we can look at the specifics of the config file.

Now back to the number one requirement for a config file: user friendliness. We know we need something that is capable to work with a complex object model. But how to phrase this?

I currently see four choices:

1. a pure natural-language look-alike like suggested by Seth Vidal (actually more a programming like type
2. a purely XML-based file
3. something like the apache config
4. something close to the way syslog-ng does things

I have to admit I am sceptic for a pure #1 approach. Natural language is intuitive for filter conditions, but it looks probably somewhat funny with most other object (which have a large number of properties). With pure XML, it is quite the opposite. It is very well with most objects, but a filter tree in XML is just plain ugly. So the most tempting choices for me are either something apache- or syslog-ng-like.

I have some mental problem with picking up the syslog-ng format, it just sound right to me. But this can be overcome. The question is if we get us into trouble when we extend the syslog-ng format and syslog-ng does a similar thing. That can become quite ugly.

An apache like format also has its drawbacks. I am not sure yet if inheritance will work well in a syslogd (but, hey, isn’t that what we currently already have?). But it sounds like a good choice. Is that something that we should model to see if any weaknesses or trouble spots occur?

Well… I can’t answer. What do you, our users, think? I would really appreciate feedback on possible config file format. I have set up a wiki page for suggestions. I would really appreciate if users could post what they would find a useful config file format.

When the object model is ready, we as the community at whole can then decide what the best possible config file format would be. If we are lucky, we’ll have a clear, easy-to-read config file format that matches the object model by winter 2007.

What I said about rsyslog versions these days may sound a bit strange. I am talking about 3.x, but yet we are only at 1.x. Do I intend to bump version numbers just for the fun of it. No! But the schema I originally had in my mind has proven to be infective. Initially, minor versions starting from .10 were reserved for devel builds, while anything below was reserved for stable ones. Consequently, we have a stable 1.0.5 right now and a devel of 1.19.0. However, the 1.19.0 is a very far departure from 1.0.5.

So I now have decided that the 1.x devel series will become 2.x stable. In fall/winter, I have a number of very large changes for rsyslog on my mind. These changes will be so considerate that the end result can not be a 2.0 version. This time I want to make this clear from the beginning. As such, the new devel series will actually start at 3.x.

So hopefully this all will make sense now: I’ll continue a bit to use the 1.19 (or higher) series for things that will become the next stable release). Around September/October, it will actually be released as 2.0.0 stable (and then receive mostly fixes). As soon as this is done, development will immediately continue with 3.x, which will be the release with new object model, enhanced threading, more plug-ins and all that…

I hope that clarifies now – and makes at least some sense to you.

Do you remember? I promised to make no more major changes to rsyslog until around September. Among the things I did not intend to do was a plug-in interface. And, yes, I did not do that. But varmojfekoj, a frequent contributor, did it ;)

As varmojfekoj says himself, this is not a full-blown plugin implementation. But he has converted the MySQL output module into a loadable plugin. The current output module interface API has some restrictions, so it is not 100% clean – but by far clean enough to be used in practice.

The big advantage of having this plug-in is that package maintainers now can create two packages for rsyslog: one for the base system and another one for the mysql functionality. This removes a lot of problems. For example, when rsyslog was compiled with MySQL support, it would have required the MySQL client to be present. This would not be acceptable for a default syslogd (after all, each system would need to have MySQL client installed…) – clearly unacceptable. Now the base (default) system can be supplied without MySQL (which is no longer tied in). And those who desired database logging, can simply add the MySQL plug-in add-on package. That package will of course pull the MySQL client, but that’s fine if you would actually like to use it (sounds only fair, doesn’t it?).

OK, finally, here is my (small) work log:

– integrated patch from varmojfekoj to make the mysql module a loadable
one
many thanks for the patch, MUCH appreciated
– added some code to support the forward-compatibilty directive
$ModLoad MySQL – this is now internally translated to the right name
Needs to be revisited, but is clean enough for now.

As promised, low activity:

– removed debian subdirectory by request of a debian packager (this is a special
subdir for debian and there is also no point in maintaining it when there
is a debian package available – so I gladly did this)
– improved overall doc quality (some pages were quite old) and linked to
more of the online resources.
– released 1.18.2

This was a question I received back from Anton Chuvakin when I asked a number of folks which features they wanted to see in rsyslog. If I could answer that question, he would reply with further details.

Actually, I think that was a very smart reply. It comes down to “why should anybody care about rsyslog at all?”. Aren’t there already enough syslogds? An interesting question.

At first, I was tempted to answer that question by talking about features, all those things that rsyslog will do to rule the world. But – does this answer the question? Nope, it doesn’t. Next, I thought to write about my goals with rsyslog. Why do I do all this work, just to create another free GPLed application. But, again, does this answer why the world needs another syslogd? Oops… no, it doesn’t. Its not about me, but about the world, that’s you, folks. So why do you need another syslogd… Continue reading “why does the world need another syslogd? (aka rsyslog vs. syslog-ng)”