thinking about a rsyslog client for Windows…

I have had a series of interesting talks during the past weeks. We at Adiscon have seen that there is high demand for closely integrating Windows machines into an rsyslog enterprise logging infrastructure. Of course, there are various ways to do that, and probably the best is using Adiscon’s other members of the MonitorWare product line. However, we can obviously go one step further and provide even thighter integration. For that reason, we will most probably soon create a special software package, the rsyslog for Windows client. It will provide

  • Event Log Forwarding
  • Log File Forwarding
  • Syslog Relay

capabilities, probably in different editions so that users can cover exactly their needs. While event log and file forwarding seem natural, syslog relay functionality may be a bit surprising, given the fact that rsyslog is available as a direct receiver. This feature is primarily targeted towards larger enterprises which may have no Linux machines in remote offices, but equipment they need to monitor via syslog. The core idea here is that we provide that functionality on a Windows box, which can than talk to the central rsyslog server via a reliable way.

We are currently discussing the details of this plan. I hope we will be able to show first results soon.

liblognorm event annotation … and CEE

As you probably know, CEE is an effort driven by MITRE to support a common event expression format. Liblognorm is a log normalizer library (aka “network event normalizer”). One of its primary target formats is CEE.

For pure normalizing needs, liblognorm extracts data fields from semi-structured log message. The extracted fields are available inside a (basically) name/value property list. Liblognorm also permits to classify messages, e.g. as being a logon or logoff message. For this classification, liblognorm provides so-called “tags”. These are simple words (strings of characters) which can be specified by the user. Tags reside in a special property called “tags”, but otherwise occupy a flat space (tags can easily be structured via punctuation).

CEE takes a slightly different approach: while it shares the tag concept (actually liblognorm inherited tags from an earlier version of CEE), CEE classifies tags into different tag types. For example, “logon” may be a tag, but can only be used to describe an action(-field). As such, “logon” can not be present by itself in a CEE log record, it must be given as value of the action field (‘action=”logon”‘). Also, CEE requires some other fields which may not be present explicitly from the original message even though the information may implicitly be present inside it. To express such information entities (and tags in the CEE way), liblognorm needs the capability to add additional fields to an  extracted event. Let call these set of fields the “annotation” for easier future reference. Liblognorm needs to annotate the event so that the target format’s (CEE) requirements are met. While I was talking about CEE so far, I assume (and know from previous experience) that other formats may have similar requirements, albeit different fields that need to be annotated.

The question is now: how to implement this in liblognorm? The initial idea was to include the annotation inside the normalization rule itself. That has a major drawback: If a rule base is to be used for CEE and some other format, the annotation may be different, and thus the same rule base cannot be used. These two rule bases would differ in just the annotation. So it seems more natural, and easier to maintain, to split the recognition rule from the annotation rule. In that setting, the message is recognized and classified by recognition rules and the annotation is based on (different) annotation rules. So only one set of recognition rules can be used by multiple annotation rules. Only the latter need to be redefined for different target formats (or systems).

This split-rule method is the way I intend to head to. In essence, the current “rule=” rule and its format will remain untouched. It will be augmented by “annotate=” rules, which contain the full annotation. The binding between these two will be done via classification (liblognorm tags): in the first step, the message is recognized, data extracted and tags assigned, just like it is currently done. Then a second step will be added. It traverses through the tags and adds all annotation that are defined for the message’s tag set. So the binding is on the tag set. Finally, it is probably necessary to add a third step that can remove unwanted fields. This step is probably target-format specific. For example, this step could eliminate the liblognorm tag set from an event if CEE compliance is desired, because CEE does not support, not even permit, an extra tag set.

Feedback on this approach is appreciated. It is my hope to be able to implement this in the near future.

filler fields in log normalization

When looking at some real-world rule bases for liblognorm, I noticed that it is often required to check for the presence of a specific field, but the value is actually not needed. This leads to fields named e.g. “filler”, “dummy”, “dummy<n>” with n being an increasing number. This is both clumsy and requires unnecessary processing power. For that reason, I have introduced “-” (dash) as field name. When this special name is used, the field as parsed as usual, but immediately discarded after the successful parse. So while we need to parse and extract in order to get the parse logic right, we save the effort to keep a copy of this unneeded data. This also means that output log records produced by the normalizer tool are cleaned up. I hope this is a useful addition.

Paper on LogNormalization

I wanted to make all of your aware that I have posted a paper on log normalization . This was originally done in regard to CEE, but I noticed that the classification of different log sources and the way to handle them is of broader use. I hope you find the paper useful.

new rsyslog config: a thank you to our users

I wanted to thank all those users who have commented during the past three (!) years on config format questions. I have consolidated all input and hope I have come up with a decent solution. Obviously, not everyone will like everything, but I hope I could find a good compromise.

So far, my blog (and the rsyslog site) has the best glimpse at the new format:

http://rainer.gerhards.net/2011/07/rsyslog-6-3-3-config-format-improvements.html

It is compatible with the old legacy format, supports simple control-flow structures (no loops, by intension) and builds heavily on name value pairs for things like actions, inputs, global settings, …

A real-world sample I used for parser development can be found at the rsyslog git web.

You’ll also find the grammar files inside that source directory in the git tree. It may be interesting for those used to flex/bison. My next step is to develop the necessary code for the name/value pair objects. That requires some more plumbing inside the core and changes to *all* loadable modules. Sounds like a lot of work, but I still hope to get this done before the summer break.

I have also started thinking about v7. It will contain a tree-based execution engine, which potentially offers even higher speed and far more options for configuration. This change is too big to make it into v6. Note that v6 will support “if .. then” and probably “if .. then .. else” but not nesting of these statements — because the rule engine does not support that. The main goal for v7 is to support nesting, including the (considerable) relevant engine changes.

I hope the new format is useful and does not upset too many. Sorry for the silence on the final selection. Past experience told me that there were too many totally conflicting views of what  the format should look like. I was deeply concerned that a broader detail discussion would have derailed this effort again. So I used known arguments and my best judgment to create the final format. Please all be assured that your arguments were deeply considered and extremely useful in getting this done.

For example, a recent mailing list discussion brought up very good argument why we actually needed to support old- and new-style config for include files. It turned out that actually the best way to solve that problem was to actually extend legacy format rather than completely replace it. This has the added advantage that textbooks, courses and a myriad of Internet-HowTos do not need to be rewritten. Besides that, I think that constructs like

mail.info /var/log/maillog

are really hard to beat in simplicity and clearness, so I think it is valuable to have them as part of the config language.

Thanks again to everyone for helping make this happen.

Rainer

rsyslog 6.3.3 config format improvements

In rsyslog 6.3.3, the config processor has finally changed. The old legacy processor (and with it the early RainerScript implementation) is thrown out and has been replaced by the so-called RainerScript processor (why that crazy name?). This is an extremely important step for rsyslog, as it now has the foundation for a much better and intuitive rsyslog.conf format. However, most of that can not be seen in 6.3.3, as it requires more work, especially in the plugin arena. Still, there are a couple of smaller improvements available.

Most importantly, the performance of script based filters has been considerably enhanced. Preliminary testing shows a three times speedup (we’ll do more benchmarking at a later stage; there is also still lots of room for optimization ;-)).

The ugliness of continuation lines has been removed. They may still be used, and this may make a lot of sense with some actions, but you are usually no longer forced to use continuation lines. Take this config snippet from a leading distro:


if ( 
     /* kernel up to warning except of firewall  */ 
     ($syslogfacility-text == 'kern')      and      
     ($syslogseverity <= 4 /* warning */ ) and not  
     ($msg contains 'IN=' and $msg contains 'OUT=') 
 ) or ( 
     /* up to errors except of facility authpriv */ 
     ($syslogseverity <= 3 /* errors  */ ) and not  
     ($syslogfacility-text == 'authpriv')           
 ) 
then /dev/tty10
& |/dev/xconsole

This can now be written as follows:


if ( 
     /* kernel up to warning except of firewall  */
     ($syslogfacility-text == 'kern')      and     
     ($syslogseverity <= 4 /* warning */ ) and not 
     ($msg contains 'IN=' and $msg contains 'OUT=')
 ) or (
     /* up to errors except of facility authpriv */
     ($syslogseverity <= 3 /* errors  */ ) and not 
     ($syslogfacility-text == 'authpriv')          
 ) 
then /dev/tty10
& |/dev/xconsole

Of course, this is not a real big advantage, but can be very useful during day-to-day operations. Forgetting the continuation marker is easy and has happend quite often, causing many more problems than necessary.

Also, the somewhat unintuitive use of “&” to chain actions together can now (optionally) be replaced by so-called blocks. For example,


authpriv.err /dev/tty10
&            |/dev/xconsole

can now be written as


authpriv.err { /dev/tty10
               |/dev/xconsole
             }

This looks much more familiar and thus intuitive to many users. Of course, both the old style as well as the new style is supported.

Finally, the need to use single quote characters (‘) over the usual double quotes (“) in script based filters was often a source of confusion. You may now use both, so ‘string’ and “string” works both. However, strings in double quotes will support parameter replacement in later versions of rsyslog. That is “Message is $msg” will evaluate to exactly this string in 6.3.3, but $msg will be resolved to the actual message content some time in the future. So be careful if you use double quotes.

Of course, none of these changes are the important ones so many users are waiting for, most importantly an intuitively-usable scoping for actions and inputs. These will be coming up shortly. I need to write some more engine code *and* need to enhance plugins to support that. I’ll probably start with actions as first. Note that the RainerScript processor already parses some of these constructs, but the rest of the engine simply ignores them. In order to get you an idea of how it will look, see this hypothetical example:


if $msg contains "error" then {
    action(type="omfwd" protocol="tcp" target="10.0.0.1:514"
           action.retryCount="-1"
           queue.type="linkedList" queue.fileName="fwdRule" queue.maxDiskSpace="1g"
           queue.saveOnShutdown="on"
          )
    action(type="omfile" target="/var/log/somelog.log")
    action(type="omuser" target="all" action.onceInterval="30")
}

I hope the example is intuitive enough to grasp it’s meaning ;) In current format, you need to write


$ActionQueueFileName fwdRule1
$ActionQueueMaxDiskSpace 1g 
$ActionQueueSaveOnShutdown on 
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
if $msg contains 'error' then @@10.0.0.1:514
& /var/log/somelog.log
$ActionExecOnlyOnceEveryInterval 30
& :omusrmsg:*

At least to me, the upcoming new way looks much nicer ;)

In regard to the distro-example given above, I’ll try to simplify it towards this form:


if ( /* kernel up to warning except of firewall  */
     hasPRI("kern.warn") and not 
     ($msg contains 'IN=' and $msg contains 'OUT=')
    ) or hasPRI("authpriv.err")
    then { /dev/tty10
          |/dev/xconsole
         }

But that’s the second step after introducing the new action statements.

Please note that the final format selection was very carefully based on many discussions both on the mailing list and inside the forum as well as needs to preserve backwards compatibility. For example, on Debian, packages drop rsyslog-specific configs into the /etc/rsyslog.d directory and expect them to be understood. In order to break things here, we needed to remain compatible with the legacy format and extend it. Only thanks to the good user feedback we could finally come up with a solution that the majority of users hopefully will find good.

With that said, I’ll now see that I create the actual release. For obvious reasons, 6.3.3 will be a bit shaky as far as the config is concerned. Most probably it will also not run the full testbench successfully (due to some very esoteric tests that are broken by actual functionality changes). However, you can be sure that the engine works well as long as it passed the config stage, because there were almost no changes during runtime (well… script filter expression evaluation has been rewritten from scratch).

rsyslog: important step to new config format

I have just merged the master-newconf git branch into rsyslog‘s master branch. With that, the new config parser becomes part of the main line. This is a very important step, as it lays the foundation to an enhanced, easier to use config format. The current version has only few enhancements, but provides the necessary plumbing to do some real nice work within the next couple of weeks. Not only as a side-effect, the performance of script-based filters has been notably increased.

I expect a release with the current state within this week. Mostly cleanup and doc work remains to be done.

Why omusrmsg is evil – and how it is fixed…

Traditional syslog files simply use the user name (or “*” for all) to send messages to users. For example, this selector will forward all mail error messages to the poor mail admin named “madmin”:

mail.err madmin

This syntax is (somewhat) intuitive, but causes severe issues when it comes to extending the configuration language. Let’s assume the mail admin is named Ian Faber and has the user name “if”. So the syslog selector would be

mail.err if

This is ok with traditional config files, but creates a problem if the language is extended. For example, rsyslog has an “if expression then” construct. Question now: how to differentiate between the user name “if” and the “if” construct? Rsyslog uses context information in order to do this. At the start of a line “if” must be the “if” construct, because “if” as a user name would require a filter in front of it. This works pretty well, but creates some complexity during config file parsing. It may also be counter-intuitive to many users. If the language is further extended (as I am doing in v6), it creates considerable extra complexity.

To resolve that ambiguity, I have upgraded omusrmsg, which handles this kind of actions, to support the regular rsyslog syntax for action configuration. You now write:

mail.err :omusrmsg:if


The extra “:omusrmsg:” tells rsyslog explicitely that an action starts and so clearly flags what the “if” stand for. This is a very vital update, and so I am extending it into all versions starting with v4. I am right now working on these changes and will release all versions ASAP. I’ll create another post when this is done. It is highly recommended to use the new syntax exclusively. The older syntax will go away in a while.

full blown dns cache for rsyslog

Up until and including version 5 rsyslog does actually not implement a real DNS cache. Instead, it uses some sort-of caching methods. They seem to work surprisingly well, as almost no real pain was reported by users in regard to this system. The big exception is UDP traffic, if combined with template options that require host resolution and a larger number of different hosts sending messages.

Starting with 6.3.1, I have now implemented a real, full-blown cache system which will resolve the issues with that use case. The initial implementation is not perfect, but I thought it would be best to gain some feedback from the community first before deciding on the final implementation. Most importantly, it currently does not expire entries (this was considered not necessary in many previous discussions we had on the mailing list). Also the current linear list data structure and locking method used is not optimal. However, it is very simple and easy to maintain. So if there is no need for more advanced (aka “complex”) code, it probably is not bad to stay simple.

I hope to get some feedback from the community, and most importantly feedback from folks who actually use the new capability to their benefit. In those cases where it matters, the speedup can be “immense”.

new rsyslog config system materializes…

The past weeks I have worked pretty hard on the new rsyslog config system. The legacy system was quite different from what you expect in modern software. Most importantly, the legacy system applied directives as the config file was read, which made it extremely hard to restructure the config file format. That also prevented features like privilege drop from working fully correct.

I have now basically changed the config system so that there is a clear difference between the config load phase and applying the config. Most importantly, this means privilege drop now works correctly in all cases (but I bet some users taking advantage of oddities of the old system will probably complain soon ;)). Other than that, there are no user-visible enhancements at the moment. However, the internal plumbing has changed dramatically and enables future change. Most importantly, this finally creates a path to a new config language, as we now have a clear interface as part of the in-memory representation of the config, which is config language agnostic.

With this initial release, there may still be some things inside the core that can be optimized. Right now, the system aims at the capability to have multiple config objects loaded (but not active) at the same time. However, there are some data instances where this is not cleanly followed in order to reuse some code. This is not a problem, because the rest of the rsyslog engine does not support dynamic config reload (and thus multiple configs at runtime) at all.

Also it must be noted that the current code is quite experimental. So there is some risk involved in running the initial 6.3.0 version. However, all dramatic changes were made to the config system. That means if the system initializes correctly, it will most probably run without any issues. The risk window is constrained to the initial startup, what should be quite controllable. Users that use privilege drop are advised to check that their configurations work as expected. The previous system did some initialization with full privileges. This is no longer the case, except for modules that actually require full privileges (e.g. imtcp to bind privileged ports). Most importantly, files are now created with dropped privileges right from the beginning. I expect that some (unclean) configurations will run into trouble with that. The good news about that is that the would run into trouble with older releases as well, but only after a HUP. Now things break immediately, what makes them much easier to diagnose.

So what’s next in regard to the config? It depends a bit on the overall workload. I will probably try to have a look at the config language next, which is another non-trivial task. Also past discussions tell me that it is extremely hard to find a format that satisfies all needs. I have already reviewed the last elaborated discussion (June and July 2010 – search for “conf” on these pages) and begun to reconsider some of the options. But this is probably a topic for a separate blog posting…