As part of project lumberjack, I am considering which tools would be vital to have in your logging infrastructure. Assume that you have a database that contains security relevant events, like syslog data or events from the Windows Event Log. Which tools would you like to have to access (and modify?) the database? Note that I am asking for small things, not something like “I’d like to have a full-blown SIEM” (of course you do). I am more focussed on tools like auditctl, tools that can be made the building blocks of your security system.
Current state of ommongodb default schema
Rsyslog’s ommongodb provides a default schema which is used for syslog data if no other is specified. It tries to align with the lumberjack project, so the schema may change in the next weeks, as hopefully a standard field set is defined there. I originally started with a very small set of fields (based on early cee/lumberjack schema), but it turned out to be too small to be really useful for real-world applications. So I have added a couple of fields today. The currently supported fields are:
- sys – name of the system the message originated from (STRING)
- time – timestamp from the syslog message (UTC_DATETIME)
- time_rcvd – timestamp when the rsyslog instance received the message (UTC_DATETIME)
- msg – the free-form message text (STRING)
- syslog_fac – the syslog facility in numerical form, see RFC5424 to decode (INT32)
- syslog_sever – the syslog severity in numerical form, see RFC5424 to decode (INT32)
- syslog_tag – the traditional syslog tag (STRING)
- procid – the name of the process that emitted the message (STRING)
- pid – the process id of the the process that emitted the message (STRING)
- level – a severity level based on the lumberjack schema definition (STRING)
Please also see my previous blog post on cee/lumberjack schema mapping, which most importantly describes the current level mapping.
Note that the default schema currently does NOT contain data obtained by parsing cee-enhanced syslog JSON part of the message. Current thinking is that we probably best include this as a sub-elements, maybe together with other structured data like RFC5424 structured data. This is currently being worked on. It’s less missing time to implement but the desire to avoid re-doing things as the spec changes. Anyhow, I’ll probably have a “timeout” after which I will implement at least some idea (after all, not too much work will be lost if things change).
If you use this schema, please keep in mind that it is experimental. At this stage I will not try to remain backwards compatible when I do changes. So excpect that newer versions may break your things!
Adiscon LogAnalyzer now supports MongoDB
I just wanted to share the good news that Andre, LogAnalyzer‘s development lead, today finished implementing a logstream driver for MongoDB. So this nice tool can now also be used to access MongoDB based data. Andre’s lab data was created by rsyslog’s ommongo output module (you currently need rsyslog git master branch to make this work). The logstream driver is not yet really optimized and we do not make full use of the NoSQL capabilites (like different schemas inside a single collection and all this). However, there is lots of exciting stuff on the todo list and I thought I mention a first successful step – and probably a quite important one if you “just want to use that thing” ;) So: good news for a Friday!
[OT] friend needs help in music contest ;)
Hi folks, my blog’s title says it is about the many thing’s Rainer is interested in. If you look at the postings, that seems to mean “syslog”. Today is one of the rare occurences that I write about something else, and it is also a request for a little bit of help…
Friends of mine produce fine (non-mainstream) music. They have entered a local contest over here with a re-interpretation of a classical theme. I really enjoy the modern “feeling” of the theme. I would like to invite all of you to listen yourself – and vote for them if you like what you hear. “Pepper Dance“, as it is called, is available form this site. Unfortunately it is in German only, but navigation shouldn’t be too hard: the play button is pretty universal ;-). And if you’d like to vote for it, just follow the big “Jetzt Abstimmen” link below the photo.
rsyslog / CEE base schema mapping
I am giving a first shot at a mapping of the CEE base schema (as currently described in project lumberjack, NOT on the CEE site!) to rsyslog properties. The core idea is to use this mapping as the default for ommongodb. Then, rsyslog shall be able to write this schema, while logtools (and others) can rely on it. For obvious reasons “rely” is not to be treated literally, as the whole thing currently is a moving target.
So I would deeply appreciate feedback for improving this mapping.
In the following mapping, the cee field name is first, the rsyslog property second.
Fields we can always map:
- srchost -> hostname
- time -> timestamp (rsyslog currently populates subseconds, what seems not to be supported in lumberjack)
- msg -> msg (initially used rawmsg, but decided against this)
- pid -> procid (may not actually be a Linux process ID)
- proc -> app-name
- level -> generated based on syslog severity (value mapping see below)
Translation of syslog severity to lumberjack level (not bijective, syslog first, number in parenthesis denotes numerical value):
- emergency(0) -> FATAL
- alert(1), critical(2), error(3) -> ERROR
- warning(4) -> WARN
- notice(5), informational(6) -> INFO
- debug(7) -> DEBUG
- (never mapped) -> TRACE
Fields we do not currently provided, but could be in some cases:
Note that these fields may or may not be present inside a JSON/BSON document.
Note that these fields may or may not be present inside a JSON/BSON document.
- ppid -> parent process ID (SCM_CREDENTIALS, local only?)
- uid -> (SCM_CREDENTIALS, local only?)
- gid -> (SCM_CREDENTIALS, local only?)
- tid -> thread ID (questionable, can probably not provided with current logging API)
As I said, feedback and suggestions are highly welcome. This list ist work in progress and can change in any instant. I’ll provide notification when the interface has stabilized. Do not expect this soon.
next steps for ommongodb
I just wanted to give you a heads-up on my work on ommongodb. During the past couple of days I have converted it to libmongo-client, which gives us a much more solid basis. I have also refactored it to some degree and adopted it to the new v6 config interface. Also, ommongodb will not be supported on pre-v6 platforms. This enables me to use the v6-exclusive features I am building now, especially great JSON and CEE support. Right now, ommongodb uses a very limited field set, and this set is hardcoded (so you can change it, but that means you need to change code).
My next step is to make ommongodb support the base event (as currently being discussed in project lumberjack). I will also provide a capability to add “extra” information from the cee field set. That’s probably not a perfect solution, but the goal is to get ready for some command line tools that are able to extract data from mongodb and thus make the system mimic it is a traditional flat-file syslog format. I have also asked Andre, the lead behind Adiscon LogAnalyzer to consider adding support for MongoDB to loganayzer. I have not yet heard back from him and don’t know exactly about his schedule, but I hope we will be able to make this happen very soon.
Only after that – somewhat hardcoded – work is done I’ll go back and look at JSON and templates in a more native way (very probably also looking at the contributed JSON string generator in more depth).
JSON and rsyslog templates
Rsyslog already supports JSON parsing and formatting (for all cee properties). However, the way formatting currently is done is unsatisfactory to me. Right now, we just take the cee properties as they are and format them into JSON format. In this mode, we do not have any way to specify which fields to use and we also do not have a way to modify the field contents (e.g. pick substrings or do case conversions). Exactly these are the use cases rsyslog invented templates for.
One way to handle the situation is to have the user write the JSON code inside the template and just inject the data field where desired. This almost works (and I know Brian Knox tries to explore that route). IT just works “almost” as there is currently no property replacer option to ensure proper JSON escaping. Adding this option is not hard. However, I don’t feel this approach is the right route to take: making the admin craft the JSON string is error-prone and very user-unfriendly.
So I wonder what would be a good way to specify fields that shall go into a JSON format. As a limiting factor, the method should be possible within the limits of the current template system – otherwise it will probably take too long to implement it. The same question also arises for outputs like MongoDB: how best to specify the fields (and structure!) to be passed to the output module?
Of course, both questions are closely related. One approach would be to solve the JSON encoding and say that to outputs like MongoDB JSON is passed. Unfortunately, this has strong performance implications. In a nutshell, it would mean formatting the data to JSON, and then re-parsing it inside the plugin. This process could be be somewhat simplified by passing the data structure (the underlaying tree) itself rather than the JSON encoding. However, this would still mean, that a data structure specific for this use would need to be created. That obviously involves a lot of data-copying. So it would probably be useful to have a capability to specify fields (and replacement options) that are just passed down to the module for its use (that would probably limit the required amount of data copying, at least in common cases). Question again: what would be a decent syntax to specify this?
Suggestions are highly welcome. I need to find at least an interim solution urgently, as this is an important building block for the MongoDB driver and all work that will depend on it. So please provide feedback (note that I may try out a couple of things to finally settle on one – so any idea is highly welcome ;)).
signifying the local host IP in rsyslog
I had an interesting enhancement request these days: make rsyslog emit a real interface address with local-generated messages (inside the fromhost-ip property). Until now, rsyslog always uses the localhost IP (127.0.0.1) when it talks of the local machine being the message source. This request makes a lot of sense and I wonder that I didn’t receive it earlier.
We started with the imuxsock module, quickly followed by imklog … and now some other internal message sources. Thinking of a local “problem” I so far have integrated the functionality into each plugin in question. It works via config directives, which enable to tell which interface’s (e.g. eth0) IP is to be used. If not specified, we default back to “127.0.0.1”, especially important for backward compatibility reasons (this change could cause a lot of harm to analysis tools!).
The current work can be found in the git v5-devel branch. However, the approach seems to be unsatisfactory: it looks like it is the right path to permit each message source to grab a real IP address. With the way it is currently done, the code for doing that needs to be replicated to all sources. That doesn’t look very smart.
So I am thinking about refactoring the refactored code ;) The core idea is that a single global function could be used to provide the local IP. Then, each plugin could query that function when it needs to know the local IP (much in the same way this currently happens for the hostname). The drawback of this approach is that only a single IP can be used across all sources. Is that a problem or a feature? Please provide comments! Also, in pre-v6 the config may be somewhat inconsistent depending on where the interface setting is made. The reason is that in pre-v6 we do not have a clear point where we know the config is fully processed (but I may somewhat work around this inside the plugins themselves). Note that this is similar to the max message size config ugliness, where the size must be set right at the top of rsyslog.conf to work properly across all modules (and this causes confusion every now and then). In v6 (6.3 and above to be precise) this is no problem due to the enhanced and much more structured config processing.
Please let me know your thoughts about this issue, so that I can take the right direction!
finally… rsyslog agent for windows released
It’s done! We have finally released the rsyslog Agent for Windows, a nice piece of software that enables easy integration of Windows Event Logs into a rsyslog backend system. Ideas for this tool floated around for roughly four to five month, and we had lots of internal discussions. It is important to note that we at Adiscon already have the necessary technology as part of our Windows products (actually, we invented this whole event-log-to-syslog type of software…), so it was just a matter of fine-tuning the code and selecting some useful default settings and policies.
The release is important because it makes clear that there actually is a Windows component (while I tried to convey that several times, people most often did not realize it due to name differences – something with “rsyslog” inside the name was expected). It is also important for me at Adiscon internally: the rsyslog Agent is a commercial product and license sales will make clear that this business is driven by rsyslog. And, obviously, I hope that this will help fund the project without need to resort to other things like premium plugins. This is also why the Agent is so important for the rsyslog project as whole: it will hopefully help to stabilize the funding situation even more.
EDIT: I should probably mention that the Windows Agent was more or less ready when I held its release in order to integrate support for cee-enhanced syslog into it. I am glad I could convince my folks at Adiscon, so that we now have this exciting feature actually available.
CEE-enhanced syslog defined
CEE-enhanced syslog is an upcoming standard for expressing structured data inside syslog messages. It is a cross-platform effort that aims at making log analysis (and log processing in general) much more easy both for log producers and consumers. The idea was originally born as part of MITRE’s CEE effort. It has been adopted by a larger set of logging stakeholders in an initiative that was named “project lumberjack“. Under this project, cee-enhanced syslog, and a framework to make full use of it, is being openly advanced. It is hoped (and planned) that the outcome will flow back to the CEE standard.
In a nutshell cee-enhanced syslog is very simple and powerful: inside the syslog message, a special cookie (“@cee:”) is followed by a JSON representation of the data. The cookie tells processors that the format is actually cee-enhanced. If you are interested in a more technical coverage, have a look at my cee-enhanced syslog howto presentation.
Adiscon is one of the main supporters of project lumberjack and CEE enhanced syslog. Since February 2012, Adiscon products offer basic support for cee-enhanced syslog, being among the first tools to do so.