Posts

Posted on

looking for Java stacktrace samples

I am currently working on log normalization as well as improvements for rsyslog’s imfile. Among the things that regularly come up on the rsyslog mailing list is support for multi-line logs and Java stack traces in general.

I would like to see what I can do to improve processing of these. To do so, I need a set of samples of such logs. As such, I look for people who would like to contribute log records for my research.

Please contact me at rgerhards@adiscon.com (or any other way you prefer) to contribute log samples. Please let me know if it is OK if I put them into the public log repository for research or you would like me to keep them private.

All types of multi-line logs are appreciated, this is not limited to java stacktraces.

Your support is greatly appreciated.

Posted on

rsyslog daily builds and tarballs

rsyslog daily builds on Launchpad

The past days, I have worked on making rsyslog daily builds and tarballs a reality. I hope this will enable users to rapidly deploy the latest features as well as make it easier to help with testing the current development system. Daily builds are what the scheduled v8-devel builds were under the previous release paradigm. Consequently, the archives are named v8-devel.

Right now, builds are only supported for Ubuntu. Users of other platforms are advised to use the daily tarballs to build from source. Depending on feedback on and success of the daily builds, I will make them available for more platforms. 

A daily build is based on the latest git master version. So it really is at the [b]leading edge of technology. So why create them?

A top reason is that I often fix a bug for someone, and that someone then is unable to build from source. In the end result, we have a bugfix, but there is no external confirmation that it really fixed the bug when I merge it into the next release. I hope that now those users can simply pick the daily build and check if that solves their problem.

Also, in general I hope that some users will use the daily tarballs to get not only the latest and greatest but contribute to the project by doing some testing.

Finally, and quite important, with daily builds we will see build problems as early as possible. In the past, we often saw problems only after source release (or very close to it), which was obviously problematic. Now, this should no longer happen. For obvious reasons, the final release build is now more or less a copy of a daily build.

As a technical side-note, daily builds are identified by the git master branch head hash that was used to build them. As a forth version component, they have the first 12 digits of that hash (an example is “8.8.0.35e7f12a2c04”). This enables us to track error reports to the right version. The packages have a different version name, based on the build date. The reason is that the hash does not increment and so newer versions (with lower hash values) are considered as “old” by Launchpad. We avoid this by using an always incrementing package version. Also note that the package changelog just contains a “daily build” entry — anything else makes limited sense.

I hope you enjoy this new feature! Feedback is appreciated.

Posted on

what’s next with rsyslog?

Now that we have released version 8.7.0, planning for 8.8.0 is in full force. I thought I’d share some of those things that made it to the top of the todo list:

I already have begun on some experimental research work on a pull model for rsyslog. Scenarios where that would have been extremely helpful surfaced on the mailing list and support forums since long. While never asked for violently, I think it is the time to explore that option. The first pilot implementation will probably very simplistic, but has a big impact: if it works for simple syslog, it will work for other pull protocols as well. That would open up some wholly new use cases. But be careful: it’s still unclear if and how fast we can realize such a method.

Secondly, we have receive a grant from GuardTime which enables us to improve the signature-related tooling. While this, too, is a bit of a large project, I will definitely begin to work on it in the 8.8 timeframe.

Finally, the ability to reparse messages is on the list. That’s another biggie, and it may be one that requires a handful of release cycles. To make this happen in a clean way, we need to change some of the internal interfaces as well as some of the processing philosophy. It will also need some good discussions on the mailing list.

Note well that these three topics won’t necessarily show up in 8.8, but at least they are something we strongly intend to work on – as said, I already started with the pull model.

Besides these three topics, there will be a number of minor improvements and bug fixes. I will also keep some focus on automated testing, but the most urgent need has been solved by the system I set up in Q4 2014. If all goes well, I’ll also get some inhouse help on expanding the testbench, what would be a real great plus.

That’s it for now, and as always: priorities may shift as needs arise ;)

Posted on

rsyslog branches and git history

There was a lengthy mailing list discussion in November and December  of 2014 of whether or not to avoid git merge entries. There was also an intermingled discussion on QA and CI. The idea was to trim the git history and make sure tests are run a quickly as possible. As a result of that discussion, I added more automated testbench runs, which also required a new branch master-candidate, which is used as a staging area to run the test, and from which changes are (manually) migrated to master when all testbench runs are OK. In order to avoid merge entries in git log, I made master-candidate the default github branch and also asked contributors to file PRs against that branch.

I’ve now tried all of this for a couple of weeks. That approach works, but it creates a lot of overhead and quite some confusion for a lot of folks. Some users have voiced they don’t really care if there is a merge entry. Fewer have voiced they don’t like them. Michael Biebl has pointed out that it is easy to make them disappear from “git log”, via the –no-merges command line switch.

After careful consideration and some frustration, I conclude that avoiding merge entries is unnecessary overhead for me. Being the 90%+ contributor for this project, I conclude that avoiding merge entries is unnecessary overhead for the project. As such, I will no longer try to avoid them at all costs. I will, however, try to keep the git history as neat as possible .. but not any more.
As such, I’ll reset the default branch on github to “master” and will accept pull requests to master. Internally, everything still needs to go through master-candidate, as this is how the new testbench setup requires. If someone doesn’t like this approach to the testbench, I am open to changes, BUT I than ask that someone to actually contribute running code to make that change happen. Good advise only is good, but doesn’t help getting things done at this stage. We already know the advise, we just have nobody who has time to implement it!

I would like to thank all users for their comments. I think they have considerable helped move forward. Sorry that I could not accept all suggestions. I guess it’s like always in life: not everybody can be fully happy. But I hope we have achieved a sufficient level of overall happiness :-)

Posted on

rsyslog -devel packages are being removed soon

If you use rsyslog’s devel packages on your system, you will receive errors soon. Be sure to read the complete posting to avoid trouble!

As part of rsyslog’s new release schedule and version naming, devel releases will no longer be named according to the “normal” numbering scheme. This also means that the previous “devel” branches will disappear, as git master branch now is the always-current devel version.

Keep on your mind that we previously had a release cycle of 3 to 9 month for a new feature to appear in a stable version. That was because new feature releases were only done when a complete devel turnaround was done, and relatively many new features were added. For this reason, some people opted to run devel versions in production, and thus needed specific tarballs (and packages) for them.

With the new six week release cycle, we get new features rather quickly into the stable builds. So it usually should be no problem to wait for the next stable to use that recently-implemented new feature. As such, there is no need any longer for special devel releases, and thus no need for devel tarballs and packages.

Well… almost. One thing I would like to have is a “daily devel version”. The idea is that if the testbench runs are OK, a new tarball and a set of packages is generated automatically and posted to a special archive. In general, that archive should receive an update once a day. So people really interested in the [b]leading edge can simply install from that daily package archive — and report bugs quickly, so helping the development process. Unfortunately, time is precious and I don’t know when and if I can setup the required automation. Most probably not before January 2015, and how it works out then needs to be seen.

In the interim, we will begin to delete the -devel packages. The old -devel tarballs will remain available, at least for the time being. The problem with -devel packages is that folks may have set their system to use the -devel repro. If we would just keep it as is, those systems would never again receive any updates, neither security-releated nor others, simply because -devel versions no longer exist in the way they were. That would pose a potentially big security risk. As such, we will delete the -devel content, and begin to do so early next week. If you use the -devel packages, be sure to switch the v8-stable instead.

Posted on

rsyslog’s new release cycle and versioning scheme

With today’s release of rsyslog 8.6.0, we start a new release schedule and versioning scheme. In a nutshell, we will be doing stable releases every six weeks now, and devel releases will be distributed via git exclusively.

We have made this move after reflecting the changes in user participation in open source development as well as analyzing what big projects like the Linux kernel, Firefox and Chrome are doing. I am very excited about the new methodology and sincerely hope it will make new features even more readily available to a large user base. Details on the new system are in the embedded presentation.

Posted on

Phasing out legacy command line options

For historical reasons, rsyslog offers a number of command line options which are actually configuration settings. These stem back to the days of the original syslogd, where the conf file was just a routing table and “all” other configuration was done via the command line. Some of them (e.g. -r to enable listening to the standard UDP port) have already been removed quite a while ago. Now, we are very serious about removing the rest of them.

The main reason is usability. The actual startup command, and thus the options, is usually well hidden in some init-system definitions. So this is highly distro specific and also heavily depends on the init system being used. It is very far from being intuitive to ask a user find that init system config option and set a specific command line option just to change a small part of the configuration — while the majority of the config stays well-defined and well-accessible in rsyslog.conf and it’s helper files.

As such, we will now either completely remove these command line switches or replace them with new configuration settings. For example, a quick poll on the rsyslog mailing list did show that nobody really cared about the -l and -s options (note the absence of replies). On the other hand, we know that options like -4 (to enable IPv4 only networking) are actually being used. For such “known in-use” options we will provide alternatives via the global() configuration object.

So what’s the schedule for this? Given the fact that we assume that most folks don’t use these options, we target a quicker change cycle than for previous options. The 8.6.0 release, due for December, 2nd 2014, will emit warning messages when these options are being used – as well as telling users they need to speak up if they need that functionality in the future. Depending on the feedback we receive, the options will be removed in 8.6.1 (Jan, 13th 2015) or a later release.

Posted on

Some cleanup in upcoming rsyslog v8

Historically, the rsyslog source tree contains a lot of seldomly-used and exotic modules. Some of them even don’t work at the moment. I kept them inside the tree so that they could serve as a sample for folks trying a similar things. However, there has been discussion on the rsyslog mailing list that all of this clutters up rsyslog and makes it a bit hard to understand which modules are well maintained, which are not, and which actually do not work or just serve an exotic border case.

I think these concerns are valid. As a consequence, I will go through the codebase and remove what is not in actual use. I will keep contributed modules which are only occasionally maintained, but I will move them to their own directory (./contrib) so that folks more easily see this is not a project-maintained plugin. Actually, we gain clarity from this move, but we don’t loose anything: if someone decides to base some new code on the then-removed code, it’s still available in older git versions. So it can still be used as a template. Besides clarity, getting rid of the cruft also eases the work of maintaining the source tree and hopefully also releases work of distro packagers.

To get you an idea of what kind of things I will remove: there are some java programs inside the code, which were used in early versions of the testbench (around v5). They are no longer in any use at all. There is omoracle, which is orphaned for quite some while, and does not work any longer since the days of v6. There is obviously no interest in this plugin, otherwise folks would have stepped up and maintained it during the past 3 or 4 years that it does not work. There is sm_cust_bindcdr, which was done as part of a custom project. While we asked for permission to include this into the project (and got it ;)), the actual module is so specific that it is extremely unlikely someone else can use it. We just integrated it as an example. These kinds of things we will remove.

Note that this step probably also helps us in moving rsyslog as whole over to ASL 2.0, which is our long-term goal since long. Some of the things now being removed (omoracle, for example) would be problematic, as they are under GPL and we cannot contact the author any longer. This is a nice additional benefit of the cleanup.

Posted on

LinuxTag Presentation now online

I realized that I had forgotten to upload my LinuxTag Berlin 2014 presentation on rsyslog enhancements and writing external plugins. I have now uploaded it, so you can view it here:

Posted on

Upcoming new v8-stable

A new rsyslog v8-stable is coming up soon. It will not just be the next iteration of 8.2, instead it will be a new feature release based on the current 8.3 devel. So be prepared to welcome 8.4. Frequent followers may wonder why 8.4 is ready. Originally, we planned to release it after the summer break. The reason is simple: its ready to come up, albeit with a little less functionality than originally anticipated.

I was (and am) busy working on the rsyslog Windows Agent, which gets a fresh brush up of its engine. It’ll be even better (and faster) as before, but that also meant that I had less time to spent on Linux rsyslog. It turned out that I am primarily doing maintenance and bug fixing on v8-devel the past couple of weeks, just as it normally happens before a new stable branch comes up. So the code has matured. At the same time, we get very good feedback for 8.2 in general, which really makes us believe that v8 fully replaces v7. The bad news is that 8.3 is currently missing the promised non-C support for input modules. However, it’s easy to do this via the regular syslog() API, so this doesn’t look like it’s overly important. In short, this means 8.3 is ready for prime time and we won’t defer it for longer than really required. Just think about how many folks have asked about non-C actions or the ability to clear out dynafiles after an inactivity timeout.

We released 8.3.3 last week, and it is scheduled to be the last 8.3 version (if nothing really important comes up). I am still working on some rough edges, which I hopefully can smoothen within the next couple of days. If possible, I’ll move them into 8.4.0. I hope to be able to release 8.4.0 next week or the week thereafter, so we get a shiny new stable before the summer break.

Also, we will finally officially drop community development support for v7. This will probably even happen this week. As usual, that doesn’t mean v7 is put into the waste bin. I’ll continue to apply patches to it, and I expect that distros will carry if for a while. Even new v7 releases may happen from time to time. But it’s no longer a version that you can expect to receive community support on (of course, rsyslog support contract customers will also be supported on outdated versions, so relax if that is you – but that’s a different story).

I hope you are looking forward to 8.4. If you can, please also help with testing 8.3.3.