Category: log analysis

Posted on

Quick update on log normalization

I just wanted to give a heads up on the status of log normalization. We have just released updated versions of libee and liblognorm. These provide important new features, like the capability to annotate events based on classification. This, among others, brings the libraries more back inline with recent CEE developments. Also, the log normalizer tool is nearly ready for prime time. The “only” thing that is missing is a decent set of rule bases. Thankfully, sagan already has a couple. I guess besides programming obtaining rule bases is a major thing to target.

As soon as I find time (I hope soon!), I’ll finalize some lose ends on the software side and get doc online on how to use the normalizer tool. I think with that a great tool for everyday use in log analysis will become available. Feedback and collaboration is always appreciated!

Posted on

A solution for invalid syslog message formats…

In syslog, we traditionally have a myriad of message formats, causing lots of trouble in real-world deployments. There are a number of industry efforts underway trying to find a common format. To me, it currently does not look like one of them has received the necessary momentum to become “the” dominating standard, so it looks like we need to live with various presentations of the same information for some more time.

The past two weeks, I have begun to make additions to rsyslog that hopefully will help solve this unfortunate situation. I know that I have no real cure to offer, but at least baby steps toward it. I have introduced so called message parsers, which can be utilized to convert malformed messages into rsyslog’s well-formed internal structure.

Why is it not a solution? Because what I really introduced was actually an interface, which permits to write different parsers for the myriad of devices. I have not provided a generic solution to do that, so the individual parsers need to be written. And secondly, I have not yet defined any more standard properties than those specified in the recent IETF syslog rfc series, most importantly RFC5424.

So why I hope this will lead to a long-term solution? First of all, there are some hopes that the IETF effort will bring more standard items. Also, we could embed other specifications within the RFC5424 framework, so this could become the lingua franca of syslog message content over time. And secondly, I hope that rsyslog’s popularity will help in getting parsers at least for core RFC5424 information objects, which would be the basis for everything else. Now we have the capability to add custom parsers, and we have an interface that third parties can develop to (and do so with relative ease).

All in all, I think this development is a huge step into the right direction. The rest will be seen by history ;) To me, the probably most interesting question is if we will actually attract third party developers. If there are any, I’ll definitely will help get them going with the rsyslog API.

Posted on

Windows 2008 Event Log…

Did you know – Windows 2008 has a much changed event log format. There are new APIs and formats all over that operating system. The first incarnation of the new logging system was seen in Windows Vista, but, being a workstation OS, it did not receive much attention from the corporate world.

When Vista came out, we at Adiscon immediately introduced the new event log monitor V2 service in MonitorWare Agent and EventReporter. That service worked well, but not many customers ever used it (who really monitors the workstation event logs…).

With the rise of Windows 2008 Server, we saw a notable increase in interest. And we finally got questions! While we always supported all properties, some of the former event log monitor properties are not available under the Windows 2008 event system. A number of customers asked if we could map them. Makes an awful lot of sense, especially if you have log analysis scripts that expect those fields. So I went to development (no longer working on Windows myself these days) and asked what we could do. I thought it would be trivial. But it wasn’t. Some mappings seem to have really hard, plus we got the impression (from lack of discussion and coverage) that we are probably among the first to ever work in this area). But nothing can stop a good programmer ;)

So I was quite happy to learn today that we have finally manage to include a full emulation of pre-Windows 2008 event properties. The code currently is in beta, and is available both for MonitorWare Agent and EventReporter.

I am especially happy as the new emulation also makes it far easier for phpLogCon to work with Windows events in a consistent way. Not to mention that I like happy customers ;)