Category: rainerscript

Posted on

multi-character field delimiters

On the rsyslog mailing list, the ability to use multiple characters as field delimiters had been requested recently. Today, I took some time off the my schedule and implemented that functionality. It is probably very useful for a number of cases. An important one is probably in combination with control character escaping, where rsyslog by default expands a single character into a four-byte escape “#ooo” with o being the octal character code (so  e.g. US ASCII HT [horizontal tab] becomes “#011”).

The new functionality is available for the RainerScript field() function. I do not intend to add it to template strings.

Some quick usage sample:

The following is the traditional way of single-byte delimiters, here with the comma character (US ASCII decimal code 44):

set $!usr!field2 = field($msg, 44, 2);
template (name=”fld” type=”string” string=”‘%$!usr!field2%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

And this is the same with the string “#011” as delimiter:

set $!usr!field2 = field($msg, “#011”, 2);
template (name=”fld” type=”string” string=”‘%$!usr!field2%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

Note that the field number (index) need not necessarily to be fixed. It can be derived from an appropriately formatted message. Here the first field contains the actual field to extract, delimiter is “#011” again:

set $!usr!idx = field($msg, “#011”, 1);
set $!usr!field = field($msg, “#011”, $!usr!idx);
template (name=”fld” type=”string” string=”‘%$!usr!field%’ — msg: %msg%n”)
action(type=”omfile” file=”/path/to/logfile” template=”fld”)

In that last sample the $msg of

“3#011val 1#011val 2#011val 32#val 4”

would return

“val 2”

Keep in mind that the first field is the field index, so the actual data fields start at 2 (field 1 is “3”, field 2 is “val 1”, field 3 “val 2” and so on).

This functionality is already present in git master head and will be released as part of 7.3.7 in the not so distant future. Some more details can be found inside the RainerScript documentation page.

Posted on

rsyslog: important step to new config format

I have just merged the master-newconf git branch into rsyslog‘s master branch. With that, the new config parser becomes part of the main line. This is a very important step, as it lays the foundation to an enhanced, easier to use config format. The current version has only few enhancements, but provides the necessary plumbing to do some real nice work within the next couple of weeks. Not only as a side-effect, the performance of script-based filters has been notably increased.

I expect a release with the current state within this week. Mostly cleanup and doc work remains to be done.

Posted on

A Web Site for RainerScript ;)

Just to make sure nobody will grab the name RainerScript, I have just created a web site for RainerScript. Of course, it does not contain any real useful information, but at least it shows I am there. As far as I know, this can be helpful in some cases ;)

Posted on

Why RainerScript?

After my initial announcement of RainerScript, I got some good feedback. I’d like to reproduce an especially useful discussion here:

> I wondered if using a complete scripting language for configuraton
> is not making it overly complex and error prone. I specifically means
> function definition etc. here.

The key is that I will model it so that the complexity is only seen when
it is actually needed. Most importantly, it will continue to understand
plain old syslog.conf format. So most peoply will probably never see the
scripting features. HOWEVER, I think it is useful to have these for a
few select folks who really need them. Function definitions, for
example, will counter syslog-ng’s feature to define a filter. I don’t
know if it is actually worth and useful to define a filter to reuse it.
But with the function, you can do it. So those coming from syslog-ng can
find their favorite feature ;)

I think it is probably worth to think about how I came to the idea of
scripting support. The root cause is complex logic, complex expressions.
A few days ago I started to implement them … and quickly found out
that actually the easiest way is “to do it right”. So I now have a
parse, bytecode and an interpreter/virtual machine running in rsyslog.
All of this “just” to support complex expressions. While I thought about
it, I finally realized that it takes just a little bit to extend these
things so that the end result will be an easy to use scripting facility.
That also solves the config file issues that’s dangeling so long and, as
a side-effect, brings up some interesting options (for those that have
need for it).

For example, I thought about how to best handle rewriting message
content. Now, the solution is obvious. I’ll implement a “set” facility
in the language that enables message variables to be rewritten on the
fly (e.g. for folks who would like to drop some parts of the message). I
agree, all not pretty mainstream, but useful in some cases. And keep in
mind that it comes at a (complexity) cost only if you need it. That’s
based on the fact that expression support was originally planned as a
by-line of the current config format, so it needed to work with the rest
in any way ;)

> But if you really need that kind of flexibility and scripting
> capabilites, have you considered to use existing languages, like lua
> , which were specifically designed to be embedded into
> applications?

Not really, because of the way it has evolved. A know a new language is
already evil, but I see big benefit for the project this time. The
reason is that I can very tightly integrate it into rsyslog’s features
like the upcoming loader. So it will be very cleanly and efficiently
integrated. For example, I don’t think that I could support old-style
format *just within* the new style format with any third-party engine.

Posted on

Introducing rainerscript ;) – and some rsyslog futures

What the heck is this? “rainerscript”? An acute case of megalomania?

Wait a moment… What’s this all about? It all starts with expression support in rsyslog and the notorious question on what the future configuration file format for rsyslog should be. From the technical perspective, I think I am finally approaching a conclusion, which will probably look quite a bit like a script language. A script language that will be specifically tailored (and well suited) to processing (system) events.

Heck, another new term… Well, if you know Adiscon’s MonitorWare family, its not new to you. In MonitorWare (where I consider rsyslog still to be a part of), we always thought about generic system events, and not syslog messages, event logs, snmp traps and all the like. The same concept somewhat naturally will apply to rsyslog and it begins to manifest with multiple input plugins. The imfile plugin is probably the first extension that brings this spirit to life. While all other plugins so far process syslog messages, imfile processes plain text files. Of course, they convert easily to syslog messages. But from a high-level design perspective you can think of event messages – some being syslog, some being file lines, some other being… use your imagination (or visit the MonitorWare site to get an idea where else events can come from).

The bottom line is that rsyslog has evolved and is further evolving into something that of course can process syslog messages. But it’s not limited to that. It can process whatever system event occurs. It’s just a matter of the input plugins. Of course, the rest of the engine must also support event classes other than syslog. That’s, among others, what I am currently working on.

To be a really useful tool, its configuration must be advanced from what it currently is. Most specifically, we must be able to change event (read: message) properties on the fly, supply different parsing capabilities, maybe have some user-supplied logic during the event flow… and so on.

The most natural solution is to provide some (more or less powerful) scripting capabilities. I am convinced that’s the right route, especially now that I have done the initial work on rsyslog expressions. It’s clean, its very flexible, its extensible and — it’s easy to use. And in order to provide freedom of choice, I’ll most probably introduce a very similar concept into Adiscon’s commercial products. And finally, over time others may find it appealing to, so I can’t outrule there will be other implementations of the same idea.

This made me come to the conclusion that I needed a better name than “rsyslog.conf” for this concept. Initially, a name of “syslogscript” came up my mind. But as I said, it’s not just about syslog. The same applies to “logscript”, we are *not* just talking about logs (at least in the future). I begun to think and after some time arrived at “EventScript”. A perfect match – but also a third party trademark (even a registered one!). So forget about it. “rscript” – also a trademark. I ran out of good names and I also became impatient. I would also like to make sure that nobody again comes along to cause grief to me, just as Huwai did with it recent junk syslog patent. So I remembered a trick a German TV journal publisher used when his competitors filed law suit after law suite over name violations of his journal when he tried to start a new publication in a field that was obviously monopolized by a few insiders: he simply used his own name in the journal title. Of course, that doesn’t make a name totally immune against legal action, but it seams to be a quite strong weapon against all these patent and tradmark trolls out there.

And so I finally arrived at the idea that a good name for the event scripting language may be “rainerscript”. Some folks (not me! ;-) ) may even abbreviate it to rscript, fishing in trademark-mined territory. Don’t do that or at least do it at your own risk ;)

I hope you find the choice of name acceptable — and the overall direction we are taking a good one. As a side-note please let me assure that rsyslog’s easy and simple sysklogd-like configuration will not go away. Rainerscript will fully support that syntax, as well as all currently existing $-configuration lines. While the previous configuration format will become part of rainerscript, you actually do not need to know anything about the scripting language unless you intend to do some pretty advanced things. Rainerscript is not something available immediately. It will evolve over time. A first glimpse at it will be available in 3.12.0 in form of the rsyslog expression support. With it, you can do things like (all of the below is on one line, which is important):

if $message contains 'test' and $facility == local7 then -/var/log/local7test.log

Notice the new if and the old style action specifier (“-/var/log…”). The script code is just a natural extension to existing configuration format.