Category: rsyslog

Posted on

on the rsyslog config file format…

A strength of rsyslog is that it support stock syslogd’s config file format. One of its weaknesses, however, is that it support stock syslogd’d config file format.

Sound strange? Jepp – but it isn’t. Understanding the old config format is great when it comes to replacing stock syslogd. It’s easy for package maintainers and it is also easy for users (who do not necessarily need to learn anything new). However, that format is clumsy, especially when compared to syslog-ng. So there is already a lot of criticism for supporting that format.

To make matters worse, rsyslog’s enhanced functionality requires some enhanced configuration directives. They have evolved as rsyslog has evolved. Currently, they are only somewhat ugly. But the more the project evolves, the more ugly they will become.

In short words: I do not like the current config file format. But on the other hand, I also do not like spending much time “just” on creating another config file parser. And I do not like the learning curve associated with it.

Seth Vidal, some weeks ago, recommended to have a directive in the config file specifying whether it is old-style or new-style. I’ll probably follow this route. That would enable us to keep backwards compatible and allow users to re-use their knowledge. In “old format mode”, features would be limited by the file format (and the users’ ability to deal with it). The full glory would only be available in “new format mode”. That sounds just fair, especially as users need to learn the new functionality and then it does not really matter if the need to learn a new conf format while they go along.

OK, that seems to be (somewhat solved).

But what format should be the new style format? Seth Vidal recommended a natural language like format (see his post on the rsyslog mailing list). However, there are many things unanswered. We don’t just have filters, we have many things.

Let’s go to the root question: what is a config file good for? In my personal view, it should

  • enable the user to specify what the product should do in an easy and simple to follow way
  • allow the application to create internal control structures that specify how the application works

Both are equally important. But the second one tells us the at least part of the config file structure is dictated by the applications object design (or call it [mnodular] structure).

So I think to do the config file right, one first needs to know the application’s object model. For rsyslog, this is not yet published. Even further: it is not yet fully designed. The 1.x versions, as well as part of the 2.x version does not have a real object design (but 2.x has at least a partial one and is somewhat like an interim release on a way to the full model).

As a side-note, this is also the reason stock syslogd and early rsyslog works with such a simple config file: it is a monolithic application and such a one is served perfectly by an equally-monolithic config file. I intend to publish a full object model for rsyslog 3.0 in the fall/winter 2007 time frame. Then, we can look at the specifics of the config file.

Now back to the number one requirement for a config file: user friendliness. We know we need something that is capable to work with a complex object model. But how to phrase this?

I currently see four choices:

  1. a pure natural-language look-alike like suggested by Seth Vidal (actually more a programming like type
  2. a purely XML-based file
  3. something like the apache config
  4. something close to the way syslog-ng does things

I have to admit I am sceptic for a pure #1 approach. Natural language is intuitive for filter conditions, but it looks probably somewhat funny with most other object (which have a large number of properties). With pure XML, it is quite the opposite. It is very well with most objects, but a filter tree in XML is just plain ugly. So the most tempting choices for me are either something apache- or syslog-ng-like.

I have some mental problem with picking up the syslog-ng format, it just sound right to me. But this can be overcome. The question is if we get us into trouble when we extend the syslog-ng format and syslog-ng does a similar thing. That can become quite ugly.

An apache like format also has its drawbacks. I am not sure yet if inheritance will work well in a syslogd (but, hey, isn’t that what we currently already have?). But it sounds like a good choice. Is that something that we should model to see if any weaknesses or trouble spots occur?

Well… I can’t answer. What do you, our users, think? I would really appreciate feedback on possible config file format. I have set up a wiki page for suggestions. I would really appreciate if users could post what they would find a useful config file format.

When the object model is ready, we as the community at whole can then decide what the best possible config file format would be. If we are lucky, we’ll have a clear, easy-to-read config file format that matches the object model by winter 2007.

Posted on

on rsyslog versions

What I said about rsyslog versions these days may sound a bit strange. I am talking about 3.x, but yet we are only at 1.x. Do I intend to bump version numbers just for the fun of it. No! But the schema I originally had in my mind has proven to be infective. Initially, minor versions starting from .10 were reserved for devel builds, while anything below was reserved for stable ones. Consequently, we have a stable 1.0.5 right now and a devel of 1.19.0. However, the 1.19.0 is a very far departure from 1.0.5.

So I now have decided that the 1.x devel series will become 2.x stable. In fall/winter, I have a number of very large changes for rsyslog on my mind. These changes will be so considerate that the end result can not be a 2.0 version. This time I want to make this clear from the beginning. As such, the new devel series will actually start at 3.x.

So hopefully this all will make sense now: I’ll continue a bit to use the 1.19 (or higher) series for things that will become the next stable release). Around September/October, it will actually be released as 2.0.0 stable (and then receive mostly fixes). As soon as this is done, development will immediately continue with 3.x, which will be the release with new object model, enhanced threading, more plug-ins and all that…

I hope that clarifies now – and makes at least some sense to you.

Posted on

rsyslog changes for 2007-08-14

Do you remember? I promised to make no more major changes to rsyslog until around September. Among the things I did not intend to do was a plug-in interface. And, yes, I did not do that. But varmojfekoj, a frequent contributor, did it ;)

As varmojfekoj says himself, this is not a full-blown plugin implementation. But he has converted the MySQL output module into a loadable plugin. The current output module interface API has some restrictions, so it is not 100% clean – but by far clean enough to be used in practice.

The big advantage of having this plug-in is that package maintainers now can create two packages for rsyslog: one for the base system and another one for the mysql functionality. This removes a lot of problems. For example, when rsyslog was compiled with MySQL support, it would have required the MySQL client to be present. This would not be acceptable for a default syslogd (after all, each system would need to have MySQL client installed…) – clearly unacceptable. Now the base (default) system can be supplied without MySQL (which is no longer tied in). And those who desired database logging, can simply add the MySQL plug-in add-on package. That package will of course pull the MySQL client, but that’s fine if you would actually like to use it (sounds only fair, doesn’t it?).

OK, finally, here is my (small) work log:

– integrated patch from varmojfekoj to make the mysql module a loadable
one
many thanks for the patch, MUCH appreciated
– added some code to support the forward-compatibilty directive
$ModLoad MySQL – this is now internally translated to the right name
Needs to be revisited, but is clean enough for now.

Posted on

rsyslog changes for 2007-08-13

As promised, low activity:

– removed debian subdirectory by request of a debian packager (this is a special
subdir for debian and there is also no point in maintaining it when there
is a debian package available – so I gladly did this)
– improved overall doc quality (some pages were quite old) and linked to
more of the online resources.
– released 1.18.2

Posted on

on reliability and the need to discard messages…

I am a bit involved in creating the next-generation on the wire syslog protocol with the IETF. Unfortunately, that activity had to starve a little while I was busy with getting rsyslog to its current state. However, I have not lost it out of my eye. And in fact the rsyslog implementation effort helped my sharpen my mind for what to suggest in the standardization area.

A recap: the IETF tries to standardize the syslog protocol for quite a while now. There were changing thoughts about what was needed, but this spring we agreed on bringing a draft up to the IESG. From there, we received a number of comments. Some easy ones, some of more substance. In July 2007, more or less one topic was left: that on what to do when a syslog sender blocks. I quote the relevant part of the IESG reply here (accentuation by me):

> First, this starts as an issue with TLS over TCP and the
> syslog base protocol.
> It can also arise teorethically for UDP, but as I understand
> not in practice for todays usage. When you are using TCP, in
> case the syslog sender generates events at an rate that is
> higher than available rate over the path used there will be
> build up of a queue. So I would like to have some words
> somewhere saying what you do when you build up a queue of
> messages that should be transmitted, but the queue simply
> keeps growing. What do I do? To me this situation implies
> that one starts discarding syslog messages and starts
> with the least important ones. So I would like to have
> a paragraph on this.
>
> I also included UDP in this in the case that you actually
> have reserved or determined that syslog is allowed to use a
> particular amount of bandwidth, but not more. In this case it
> could be possible that one implements a rate limiter
> and run into exactly the same issue as for TCP.
>
> Please do understand that if syslog was designed from scratch
> today it wouldn’t get away without a congestion control that
> guarantees that it isn’t missbehaving in any situation. But
> being an “old” protocol we are accepting less than that.
> However, we do require it to contain the limitations and
> assumptions that it can be safely operated with. Using it as
> it currently is used is not an issue because the networks it
> is used in has many magnitudes more bandwidth that syslog
> generates. However, what would happen if some one starts
> using syslog in low-powered, low-bitrate sensor network for
> carrying some information. In that situation syslog becomes a
> potential threat to the stability of the network as it
> doesn’t react at all to congestion if run over UDP. Network
> failures are also sitation that are problematic to ensure
> that the inteded resources are available. Thus we do
> like to protect the utility of what resources do exist.

And this sentence covers it all:

> Please seriously consider adding a paragraph about
> how one can thin a queue of syslog messages in
> the base protocol. This as I think it potentially applies
> to any transport.

This basic requirement caused some confusion in the working group (WG). I have to admit, it confused me at that time, too.

In the mean time, I kept working on rsyslogd. I didn’t even realize that I had written code in 2005 (or so) to take care of what was requested by the IESG: if rsyslog runs in single thread mode and it forwards messages via TCP and the send call would block – then we have a problem. Because when we wait, we will potentially hang the whole system. This is unacceptable. So I coded rsyslog in such a way that it discards the message in such a situation.

When I moved rsyslog to a multi-threaded design, that restriction was removed. Now, there was a queue between the receiver and the (potentially blocking) sender. And, as we all know, queues are endless, so we’ll never run into problems ;-]. Of course, that simplistic point of view survived only so long as we did not deliberately bash rsyslogd with lots of traffic. What then happens is pretty much the same as in the single-threaded case: rsyslog freezes. Mildew, a frequent contributor, discovered this problem and also sent a good fix: he throws away messages if the queue is full. Sounds familiar? Yes, there seems to be a real-world problem and there is also a well known cure for it.

In some essence, this is what the IESG is asking for – at least as of my understanding. But the IESG, obviously speaking with a lot of experience, goes a step further. What they recommend is not to discard newly arrived packages. The recommend a way to “thin out the queue“. So we would not just blindly drop whatever comes when the queue is full. No, we should intelligently decide what to drop, and that based on the severity of the message. It sounds only fair that debugging messages get dropped in favor of emergency messages. And, yes, syslog has a field “severity”, which is the right thing to use here.

I thought about implementing a “high watermark” in the rsyslog queuing engine. What that amount of the queue would be full, a filter should be applied to the message. So from that on, only messages with a given severity would be queued, while all others be dropped. Of course, those already in the queue would be sent, no matter how unimportant they are. So far, I have not implemented that algorithm. And I think I am now unlikely to do it that way. It would be much better to follow the advise and, once the queue becomes full, begin to drop messages based on their severity.

I can even envision the algorithm: go forward through the queue starting at the tail and look for messages with the least importance. Once you have found one, drop it. Then enqueue the (new) higher-severity message. When this is done, do not drop any more messages, because it may not be necessary. We can always drop if a newer higher priority message arrives. If, during our queue traversal, we could not find any message with lowest severity, retry with the next lowest severity. Run this algorithm until either a message to drop is found or the drop-severity has reached the severity of the to-be-queued message, which in this case is the youngest and thus the drop target. Of course, that algorithm needs to be tweaked in regard to performance, but basically it should work fine. I even think, a derivative of it will make it into a rsyslog feature request.

… and now back to the IETF. We still need wording for our upcoming standard (called an I-D). Now that I know what I look for, shouldn’t it be easy? Well, to be honest I don’t think its that easy. First of all, I am not sure if, in a protocol specification, I can demand such queue processing. It looks somewhat too algorithmic, too implementation specific to me. So I need to find a better way to phrase it. And of course, peer review will show if that makes sense or not (maybe we even get a better suggestion).

My first try at it is as follows:

In any implementation, a situation can arise in which an originator or relay would need to block sending messages. A common case is when an internal queue is full. This might happen due to rate-limiting or slow performance of the syslog application. In such cases, it is RECOMMENDED that the originator or relay drops messages of lower severity in favor of higher severity messages. Messages with a numerically lower SEVERITY value have a higher severity than those with a numerically higher one. To-be-dropped messages SHOULD simply be discarded. The syslog application may notify a collector or relay about the fact that it drops messages. If the underlying protocol supports congestion control, discarding of messages SHOULD be avoided.

I have to admit that this text is probably too lengthy and imprecise. But hopefully it is a starting point. I’ll post it to the syslog IETF WG and we’ll see…

Still, there is one important message to learn: no matter how hard we try, there is always a chance that we must discard messages. It may be highly unlikely and it may be totally unexpected. But we can not absolutely avoid it. Not even an on-disk queue is large enough, that it can buffer any imaginable message queue. So loss of messages is inherent in any application, also in syslog and auditing. The question is, how we deal smartly with it and how we preserve as much evidence in our logs as possible (and keep the log valid in those cases!). This all sounds pretty basic and easy, we always need to remember ourself’s of this fact…

Posted on

why does the world need another syslogd? (aka rsyslog vs. syslog-ng)

This was a question I received back from Anton Chuvakin when I asked a number of folks which features they wanted to see in rsyslog. If I could answer that question, he would reply with further details.

Actually, I think that was a very smart reply. It comes down to “why should anybody care about rsyslog at all?”. Aren’t there already enough syslogds? An interesting question.

At first, I was tempted to answer that question by talking about features, all those things that rsyslog will do to rule the world. But – does this answer the question? Nope, it doesn’t. Next, I thought to write about my goals with rsyslog. Why do I do all this work, just to create another free GPLed application. But, again, does this answer why the world needs another syslogd? Oops… no, it doesn’t. Its not about me, but about the world, that’s you, folks. So why do you need another syslogd… Continue reading “why does the world need another syslogd? (aka rsyslog vs. syslog-ng)”

Posted on

a wiki for rsyslog

A modern open-source project, at least IMHO, needs a number of supporting web services: a blog, a bug tracker, a forum … and a wiki. For rsyslog, most of this services existed for quite a while. What was missing until now was a wiki. Thanks to Andre Lorbach who set it up, we now have one at http://wiki.rsyslog.com. I have finally found time to add at least some information into it.

And now it is the users turn. The wiki is not meant to be a replacement for the official doc. Instead, it should contain user contributed content and ideas. So it is up to you to fill it with live. Of course, the rsyslog devel team will have a keen eye on the wiki and be as helpful as possible. Please give it a try and help creating a great community-driven wiki.

Posted on

rsyslog changes for 2007-08-09

Yesterday I made a couple of changes:

– fixed a bug in outchannel code that caused templates to be incorrectly
parsed
– fixed a bug in ommysql that caused a wrong “;template” missing message
in some cases
– some text and code cleanup in cflineParseTemplate() to match the changed
design of selector handling
– added new doc files to autotools for make dist
– fixed a bug that would have manifested when module configuration commands
would have been freed. The current code base, however, never does this, so
this is not a real bug fix – it’s in preparation of future work
– fixed a potential memory leak in modules.c, again, this could not happen in
current code
– added interface API for unloading module (a dummy)
– added module unload functionality; rsyslogd now unloads modules on exit
(of course, with only statically linked modules, there is little current
value in this – but it is made towards an upcoming dynaload plugin interface)

Posted on

rsyslog changes for 2007-08-07 & 08

This time, I blog the changes for both days. Today was again quite busy, with a number of smaller changes. But all in all you can see that the pace of changes is reducing. I hope that people will test it, but I also hope that we will see no more big bugs the next few weeks. Then, I consider it to be mature for a new stable release.

You will probably see no or very few code changes during that period. You can expect, however, some doc upgrades. All in all, my focus has now moved to the next major release, which I am currently designing. And of course, some other things are also asking for a bit of time now that I have worked very intensely the past two month on rsyslog.

Stay tuned. Any news will appear here in the blog. In the mean time, try 1.18.1 and tell us what needs to be fixed (of course, you may also just praise our work, that’s fine, too ;)).

Here is the work log:

2007-08-07

  • applied patch from Michel Samia to fix compilation when the pthreads feature is disabled
  • moved code to create/delete message queue to init(), after reading the conf file. This is a prequisite to allow specifying the queue size in the conf file.
  • add config directive $MainMsgQueueSize, which now allows to configure the queue size dynamically

2007-08-08

  • moved the “after select polling code (for fds)” to its own function
  • optimized select handling, after select polling cycle is now finished when all active selectors are processed
  • all compile-time settings are now shown in rsyslogd -v, not just the active ones
  • changed function name dprintf() to dbgprintf() as it conflicts with the clib
  • added config file directive $ActionResumeInterval
  • solved an issue with debian sid. LARGEFILE preprocessor defines are changed in rsyslog.h, which causes grief for zlib under sid. As a temporary solution, I have moved rsyslog.h right at the beginnng of the include order. It’s somewhat dirty, but it works. I think the real solution will be inside the autoconf files.
  • added a database cleanup script based on a suggestion of Michael Mansour. Michael Meckelein adapted it slightly for use with rsyslog.
  • released 1.18.1
  • fixed bug: default for $DropMsgsWithMaliciousDnsPTRRecords was not reset on up and on $ResetConfigVariables
Posted on

next generation rsyslog design, worker pools, parellelism and future hardware

Based on my last post on rsyslog multithreading, an email conversation arose. When I noticed I elaborated about a lot of things possibly of general interest, I decided to turn this into a blog post. Here it is.

The question that all this started was whether or not a worker thread pool is good or evil for a syslogd.

Yes, the worker-thread pool has a lot of pros and cons with syslog. I know this quite well, because I am also the main design lead behind WinSyslog, which was the first syslogd available natively on Windows (a commercial app). It is heavily multi-threaded. I opposed a worker pool for a long time but have accepted it lately. In some high-volume scenarios (unusual cases) it is quite valuable. But of course you lose order of messages. For WinSyslog, we made the compromise to have the worker pool configurable and set it to 1 worker if order of events is very important. I designed WinSyslog in 1995 and released the first version in 1996 – so I know quite well what I am talking about (but to be honest the initial multi threading engine got in somewhat later;)).

HOWEVER, especially in high-volume environments and with network senders, you are somewhat playing Russian roulette if you strongly believe that order in which events come in is exactly the same order in which they were sent. Just think routers, switches, congestions, etc… For small volumes, that’s fair enough. But once the volume goes up, you do not get it 100% right. This is one of the reasons I am working quite hard it the IETF to get a better timestamp in into syslog (rsyslog already has it, of course as an option). The right thing to do message sequencing is by looking at a high-precision timestamp, not by looking at time of reception.

For rsyslog, I am not yet sure if it will ever receive a worker pool. From today’s view, it does not look necessary. But if I think about future developments in hardware, the only key in getting more performance is by using the enhanced parallelism the hardware will provide. The time of fast single cores is over. We will have relatively slow (as fast as today ;-]) massively parallel hardware. This is a challenge for all of software engineering. Many folks have not yet realized it. I think it has more problem potential than the last “big software crisis”. As you can see in the sequencing discussion, parallelism doesn’t only mean mutexes and threads – it means you need to re-think on how you do things (e.g. use timestamps for correlation instead of reception/processing order, because the later is no stable concept in a massively parallel program). With the design I am now doing, I will definitely have separate threads for each output action (like file or database writer). I need this, because rsyslog will provide a way to queue messages on disk when a destination is not available. You could also call this “store-and-forward syslog”, just like SMTP is queued while in transit. You can not do this concept with a single thread (at least not in a reasonable complex way). I also do not think that multiple processes would be the right solution. First off, they are too slow. Secondly, multiple processes for this use are much more complicated than threads (I know, I’ve written similar multi-process based things back in the 80s and 90s).

Rsyslog will also most probably have more than one thread for the input. Input will become modular, too. If I use one thread per input, each input can use whatever threading model that it likes. That makes writing input plugins easy (one of my goals). Also, in high volume environments, having the ability to run inputs on multiple CPUs *is* *very* beneficial. One of the first plugins will be RFC 3195 syslog, which is currently run as an external process (communicating via unix local domain socket, which is quite a hack). At the core of 3195 is liblogging, a slim and fast select-server I have written. However, integrating it into a single-threaded application is still a challenge (you need to merge the select() calls and provide an API for that). With multiple threads, you can run that select server on its own thread, which is quite reasonable (after all, why should both UDP and 3195 reception run on the same thread?). The same is true for tcp based processing. Then think about the native ssl support that is coming. Without hardware crypto acceleration, doesn’t it make much sense to run the receiver on its own thread? Doesn’t it even make sense to run the sender on its own thread?

So the threading model of the next major release of rsyslog (called 3.x for reasons not yet elaborated about) will most probably be:

  1. multiple input threads, at least one for each input module (module may decide about more threads if they like to)
  2. all those input threads serialize messages to a single incoming queue
  3. the queue will most probably be processed by a single worker thread working on filter conditions and passing messages to the relevant outputs. There may be a worker pool, but if so its size can be configured (and set to 1, if needed)
  4. multiple output threads, at least one for each output. Again, it is the output’s decision if it runs on more than one thread
  5. there may be a number of housekeeping threads, e.g. for DNS cache maintenance

This design will provide superb performance, is oriented on logical needs, allows for easy intrgration of plugins AND will be reasonable easy to manage – at least I hope so.

But, wait, why have I written all this? OK, one reason is that I wanted to document the upcoming threading model for quite a while and now was a good time for doing so. But I think it also shows that you can not say “multithreading is good/bad” without thinking about the rest of the system design. Multi-threading is inherently complex. Humans think sequentially, at least when we think about something consciously. So parallel programming doesn’t match human nature. On the other hand, nature is doing things itself massively parallel. Just think about our human vision: in tech terms, it is massively parallel signal processing. This is where technology is also heading to. As we have learned to program at all, we will also learn to deal with parallel programming. A key thing is that we should never think about parallelism as a feature – it’s actually a tool, that can be used right or wrong. So let’s hope the rsyslog design will do it right. Of course, comments are very much appreciated!