phpLogCon becomes Adiscon LogAnalyzer

I have blogged the past days about Adiscon LogAnalyzer. We are now gradually rolling out the new site. So I thought it is a good idea to reproduced my “official announcement” on the blog as well:

As in all things, there is a certain fashion in open source project names as well. For a long time, “php*” was a great name for php-based open source solutions. However, nowadays these somewhat bulky names have been replaced by “more streamlined” names.

I personally think that dropping the “php” part makes it somewhat easier to speak and write about these projects. So we decided it was right to drop “php” from “phpLogCon”. But was “LogCon” the ultimate name for a tool to search, analyze and (starting with v3) report on network event logs? A quick discussion within our group as well as with some external buddies turned out that “LogCon” is probably pretty meaningless. Even if one deciphers “Con” for “Console” – what does it mean to be a “Console” in this context? Not an easy to answer question. Bottom line: “LogCon” is pretty meaningless.

So we thought we do “the right thing” and rename the project before it becomes even more widely spread. The later you do a name change, the more painful it is. That made us think about good names. We ended up with “LogAnalyzer”, because analysis is the dominant use case for this tool (especially if you think of reports as being part of the analysis ;) ). Another quick search made us aware that there are (of course) lots of “LogAnalyzers”. And, of course as well, all second level domains where taken.

Bare of an expensive legal adviser, we made the decision to boldly name the project “Adiscon LogAnalyzer“, aka. “the log analyzer (primarily) written by Adiscon”. With that approach we use our company name (which obviously legally belongs to us) together with the generic term “LogAnalyzer”. That is done in the hope that it will resolve any legal friction that otherwise may occur. For the very same reason you will see us consistently referring to “Adiscon LogAnalyzer”.

We are aware, however, that this implies some other cost: A project with a company name inside it does sound a bit like a purely commercial project. On the other hand, that seems to be no problem with the big players, like “Red Hat Linux” or “SuSe Linux”. So we hope that the company part inside the name will not have a too-bad effect on this project.

We pledge that Adiscon LogAnalyzer will always be a free, open source project. And the GPLv3 we use is your guarantee for that.

In addition to the core Adiscon LogAnalyzer, Adiscon will also provide some non-GPLed components in the future. And we hope that others will do that as well. Our sincere hope is that Adiscon LogAnalyzer will evolve to a framework where many third parties can plug in specific functionality. Consequently, we have added a plugin directory to the new site, and some third-party written message parsers already populate it.

So – phpLogCon has not only a new name and a new site, it is also more active than ever and eager to solve the log analysis and reporting needs for a growing community. Please help spread the word!

is a third-level domain suspect to google?

If you follow this blog, you’ve probably already heard that we are doing a name change for phpLogCon: it will soon be known under the name Adiscon LogAnalyzer (with the Adiscon in front of the “real” name to ease potential legal issues).

Among others, that means we need to change the web site. Not surprisingly, no second-level domain with loganalyzer in it was available at the time we searched. Most of them, of course, been taken by domain spammers. So we settled for the loganalyzer.adiscon.com name. As I found out yesterday in Google Webmaster Tools, that this may cause some troubles. Google provides a “Change of Adress” tool that is meant to be used in situation.

However, I discovered that this tool does not work with third-level domains. All I see when I try to use that tool is the message “Setting is restricted to root level domains, only” (as a slight technical side-note, it should say “second level domain” as I don’t think it works for com, net, org, … only ;)).

While browsing the google help forum, I found that others seems to have similar problems. For example, people in the UK, where everything is a third-level domain (for example, .co.uk is what .com is for the international Internet).

Given this stance, I wonder if google punishes third-level domain sites in any other way. If so, our decision to move to the new site may not be a good one. I have posted a question in the Google help forums. I guess I will not get a definite response, but maybe one can read between the lines.

I will keep you posted, also on the overall progress of the name/site switch. We have now entered the “hot phase”, meaning that we actually intend to roll over to the new site within the next couple of days. Stay tuned for more news and more features.

new phplogcon site

Today, I received a first more or less complete link to what will become the new phplogcon site. The site is not yet live, but will provide some of the new features.

If you look at it, you’ll probably notice a couple of things. First of all, the name “phpLogCon” is no longer spelled out. The reason is that we considered a bit bulky and meaningless. “Loganalyzer” is exactly what the tools is about. But, of course, there are a myriad of (trademark) problems related to that name. So we try to avoid all confusion by calling it “Adiscon loganalyzer”, hoping that the company name as dominant part of the product name will rule out all problems. For that very reason, you’ll also see me to refer to Adiscon Loganalyzer in the future. If you wonder why I stress that “Adiscon” part, you now know why.

Secondly, you will notice the fresh design. While I am not a visual guy, I have to say that I like it very much. I think it removes much of the clutter and makes it easier to find the information you need quickly. We also have changed the content management system in the background. The new sites uses WordPress, which seems to be highly approprioate for what the site needs. Of course, the wiki and forum will remain as they are – they have proven to be quite well as they are.

If you look more closely, you will also note that Adiscon LogAnalyzer gets an important new component: a reporting module. I managed to convince my peers at Adiscon to move some of our MonitorWare Console closed source technology into Adiscon LogAnalyzer. My long-term vision is that reporting capabilities will much enhance the utility of this tool. In order for Adiscon to get something back, we will begin to develop some enhanced reports, which will be non-free for commercial users. However, the base product as well as some base reports, will always remain free!

I hope you consider this to be good news, just as I think! Thanks to everyone who made this possible.

rsyslog – what’s next?

I’ve not blogged so much the recent weeks. I have had my nose deep down inside rsyslog code, adding new features (like an automatic zip file writer or the ability to spoof/forge UDP sender addresses) and enhancing performance (where I think I scored some major points ;)).

So it is time for an update. Where is rsyslog heading to? With the many changes I made in the past two to three month, I think it is very important to let the code base stabilize. So I would prefer not to touch too much existing code for a while. Also, it is summer time and my summer vacation is not so far away. Another good argument that it is probably not the best time for big code changes (or do I like to break things before I go away…? ;)).

So looking at what to do next, I would like to center myself on improving the tool set that helps create rsyslog. That doesn’t mean direct improvement to the actual syslogd, but rather to tools that help build and maintain it. The first major effort in this regard was adding an automated testbench. If you look at v3, I think it has around four automated tests (previous versions had none). With v5, we have over 20 subtests, each of which test various cases, so in total we currently have around one hundered test cases automatically covered.

When I started with this in v3, it was a major effort, even though the number of covered cases was rather small. But getting started with a testbench meant I needed to evaluate ways to automate the tests and create them in the best possible way for rsyslog (which also means convenient during the development process). At that point, I tried a lot of things and finally came up with the current set of tests. The initial testbench covered only a very limited set of use cases.

Since then, it has greatly improved, but there are still a lot of uncovered areas. But I now regularly add new tests, most often when I implement new features, change existing ones or hunt bugs. The process is now well understood and many tests can be added with relative ease (but others not, I have some testcases in the queue that require notable extensions to my current system plus a bit … of the different toolset I will be talking about soon…).

Initially, I was rather skeptic if the testbench would really pay, especially after I saw the initial effort required (which I by far underestimated). But in the meantime I am convinced. Especially the past couple of month has shown that the automated tests both increase development productivity (by reducing the number of manual tests that need to be done and spotting regressions early) as well as code quality (detect regressions where they otherwise would have been overlooked).

Now I am in a similar situation in regard to performance testing as I was in regard to correctness testing: everything is done manually and with very low-level tools. Still, I was able to make good progress without tools. But I hope that tools would be as useful for performance testing as they were for bug hunting. Most importantly, my current performance improvement testing covers only limited (though highly relevant) scenarios: those where getting sufficiently reliable numbers is possible with the limited capabilities I have. Most importantly this means that almost all testing so far has been done with plain tcp syslog. While this still enables to check the core engine’s performance, it does not offer a clear view of e.g. UDP performance (which I really do not have now). Also, the examples are artificial, and it would be useful to get more of a real-world performance benchmark.

Finally, performance benchmarks stress the engine, especially its multi-threading capabilities. So performance testing is also a good way to uncover those nasty threading bugs, that one otherwise only detects when systems fail in production (and nobody then knows why…). So I consider decent performance test also to be a plus for code quality. I even consider them very important to stability e.g. the v5-engine, which so far has received only limited attention in practice. It looks like almost nobody ever tried it. I know because the initial v5 release had such a big memory leak, that any serious tester would have needed to come up and complain very quickly. A lack of test deployments makes it harder to mature the engine. I think that good stress tests (which all have a performance co-notation) will help to somewhat mitigate this problem. As a side-note, I have uncovered many of those bugs that I fixed during my manual performance testing. This seems to prove the point.

So I am more or less convinced (if nothing more urgent shows up) to spend some time implementing performance tools and tests for rsyslog. I would also like to include a somewhat older idea of a “diagnostic front-end” that would be able to pull (and maybe modify) some of rsyslog parameters. I’d expect that as a side-activity I’ll also gather (at a minimum) or improve (preferable) performance in a couple of areas (UDP reception performance is on top of the list). But improvements will only come after the basic tools have been written.

As with the testbench, that will mean that new features and enhancements will probably stall a bit in the coming weeks. This even more as I do not intend to write the front-end in C (I personally do not consider C to be the language of choice for non-performance critical interactive programs, especially looking into some of the portability issues – but YMMV…). So I will try to approach this with an Java app. I have to admit that I learned Java 8 to 10 years ago and never programmed much in it, so that will probably mean I’ll need to re-learn the language, but as I don’t consider this GUI to be something extremely critical, I don’t see any issues with me as a Java freshmen doing it.

As a side-note, I should probably mention that I am also involved in the phpLogCon project. So far, I am only part of the design team, but I have a number of really cool visualization features on my personal wish-list. If I ever get time to work on that (I hope for next year), I probably will need to do that in Java, so it doesn’t hurt to practice on a less demanding project. In that sense, I also hope to be able to set stage for some future cool technology while I work on a current demand ;) It would also be interesting so visualize some of the performance counters, but that’s another story ;)

All in all, getting an interactive troubleshooting and analysis front-end has big potential, not only for testing but also for deployments and finding configuration bugs (which become more and more an issue with improving complexity of the configuration). One could also envision that it could include a graphical configuration build … as well as tools for setting up all those TLS certs. I don’t think I can do all of this now or in the next quarter. But I think it is the right time now to begin working on a foundation that offers yet another big potential. Especially as it also helps to urgent need to get better testing for the engine plus the desire to further improve its performance (my goal is no less than to provide the by-far fastest AND most reliable syslogd on this planet ;)).

Well, that’s it for now. I hope you like the idea of an additional performance-centric toolset (which of course also requires engine enhancements) and a GUI as much as I do. If you have comments or concerns, please let me know. I sincerely hope to begin a new round of capability enhancements with this move.

SyslogAppliance 0.0.6 out

Finally, I made a new version of the syslog appliance. It is not the really big release. Friday, I intended to do just a refresh, but then I ended up integrating a capability to discard messages older than 60 days (obviously an optional feature). Still, it looked like a quick action, but phpLogcon gave me a somewhat hard time. Finally, I even discovered a bug and could fix it.

Probably the next milestone for the appliance is SMP support. I’ve done some preliminary work, but on the other hand there is so much more to do. Let’s see…

a logging appliance

The IT world is increasingly turning towards appliances: pre-configured systems, which do exactly one job an do it well. Like a household appliance, all you need to do is plug it into your infrastructure, maybe change a setting or two and you are ready to go. While previously appliances always had a co-notation of a hardware box being delivered, the increasing popularity of virtualization enables to build pure software appliances.

One of the things we intend to investigate is create a logging appliance, using VMWare tools. We will set up a standard Linux, use MySQL and Apache with it and install rsyslog and phpLogCon. That in a “ready to use” fashion where only the devices need to be pointed to the right IP address of this virtual box.

This is one of my next projects and feedback on such an effort is very much appreciated.

rsyslog work log 2

As you can see below, I am currently more involved in non-coding activities, like getting most reliability out of plain tcp syslog, IETF work and pondering the phpLogCon team with all those nice little things that are hard to implement ;)

Past day’s rsyslog work log:
2008-05-28
– released 3.17.3
– (finally switched to Fedora 9 on main dev machine;))
2008-05-29
– enabled Posix ERE expressions inside the property replacer
(previously BRE was permitted only)
– provided ability to specify that a regular expression submatch shall
be used inside the property replacer
– Worked on Martin’s suggestion for a more reliable plain tcp transport
2008-05-30
– implemented in property replacer: if a regular expression does not match,
it can now either return “**NO MATCH** (default, as before), a blank
property or the full original property text
– Worked on Martin’s new suggestion for a more reliable plain tcp transport
This one looks promising, at least for simple cases
2008-06-02
– worked on plain tcp syslog reliability
2008-06-03
– begun working on TLS doc (finally ;))
– bugfix: part of permittedPeer structure was not correctly initialized
thanks to varmojfekoj for spotting this
2008-06-04
– bugfix: off-by-one bug during certificate check
– bugfix: removed some memory leaks in TLS code
– enhanced property replacer to support multiple regex matches
2008-06-06
– IETF work, syslog-transport-tls
– begun doc on unreliable tcp syslog and a work-around based on Martin’s
idea

The MonitorWare Knowledge Base

There’s currently a lot brewing over here. With the release of phpLogCon, I finally got to a stage where we have a decent web front-end that enables us to support admins with troubleshooting right while they look at their log data. Of course, such a system not only deserves careful design, but it requires a knowledge base so that people troubleshooting can find solutions.

If we look at our web sites, one notices there are lots of troubleshooting resources already available. Just have a look at forum.adiscon.com and you see what I mean. This forum not only offers product specific support, but has a number of quite generic discussion forums (covering Windows Events and syslog messages). We also have other troubleshooting databases. Then, each product site (like rsyslog) has its own support forum.

While all of this is great, it does not play well with the idea of the central, one-stop troubleshooting resource we intend to build. So our first step towards this resource is putting the existing resources under a single umbrella and placing them into a single system. That, of course, requires some redesign and won’t be perfect from day one.

The initial step is to consolidate all forums into a single one. My friend Andre is right now doing that. He has set up the new site kb.monitorware.com which in the future will provide all troubleshooting resources in an easy to find way. That site will be highly integrated with phpLogCon, which will be able to pull troubleshooting info from the central repository while looking at the local logs. I am very excited in seeing this become a reality (though I have to admit we are several month away from the ultimated goal).

The knowledge base site is currently in experimental operation and being finalized for production use. I hope to be able to go officially online next week.

finally… phpLogCon v2

Finally, we have released phpLogCon 2.1.0, the first official beta version of the v2 branch. PhpLogCon has been completely rewritten from scratch. It now offers a state-of-the art modern user interface and also is able to work with log files and not just databases. For example, it can be used to view a remote server’s log files over the web (proper authentication settings highly recommended). It will evolve to a very capable search, reporting and analysis frontend for syslog data.

Let me stress the point that it can work with log files directly. For example, we have set it up on one of our mail relays so that we can review mail logs without the need to login onto that machine. Obviously, this functionality should only be available to authenticated users, but then it is quite useful. I would appreciate to learn about any more thought of how this tool can be put to good use.


Download
:

We are currently setting up the infrastructure (mailing list et al) for phpLogCon. I’ll do one more announcement here when this is completed. In the meantime, I suggest subscribing to freshmeat’s announcements, which we maintain in a timely way. You can do so at:

http://freshmeat.net/projects/phplogcon/

phpLogCon – why the next version

I am in an interesting email discussion and would like to share something on phpLogCon that’s probably of interest for others, too:

A major reason for phpLogCon v2 is enhanced functionality. We’ll switch away from a pure database paradigm. We’ll be able to work with log files (much faster and sufficient in many cases), of course databases but in the long term also with a specialized not-yet-written logging-specific datastore. We also want to build a community around the new phpLogCon and as an important step set up a public troubleshooting database and of course connect it to exisiting ones on the web. So the core idea of phpLogCon v2 is to create a system where users can analyze their logs but at the same time collaborate on finding solutions to issues they see (in the long term we may even get to a point where we can identify problems based on patters – but that is far too far away ;)). So the phpLogCon idea has gotten a bit broader (I should probably post that on the side, too).

Of course, rsyslog will also contribute to this vision – the overall idea is to create a great system monitoring and auditing system that not only helps with compliance but enables you to fix any upcoming trouble before it really hurts.