rsyslog Archives - Page 5 of 41

2014-12-02

rsyslog’s new release cycle and versioning scheme

With today’s release of rsyslog 8.6.0, we start a new release schedule and versioning scheme. In a nutshell, we will be doing stable releases every six weeks now, and devel releases will be distributed via git exclusively.

We have made this move after reflecting the changes in user participation in open source development as well as analyzing what big projects like the Linux kernel, Firefox and Chrome are doing. I am very excited about the new methodology and sincerely hope it will make new features even more readily available to a large user base. Details on the new system are in the embedded presentation.

Rsyslog version naming (v8.6.0+) from Rainer Gerhards

2014-11-27

Phasing out legacy command line options

For historical reasons, rsyslog offers a number of command line options which are actually configuration settings. These stem back to the days of the original syslogd, where the conf file was just a routing table and “all” other configuration was done via the command line. Some of them (e.g. -r to enable listening to the standard UDP port) have already been removed quite a while ago. Now, we are very serious about removing the rest of them.

The main reason is usability. The actual startup command, and thus the options, is usually well hidden in some init-system definitions. So this is highly distro specific and also heavily depends on the init system being used. It is very far from being intuitive to ask a user find that init system config option and set a specific command line option just to change a small part of the configuration — while the majority of the config stays well-defined and well-accessible in rsyslog.conf and it’s helper files.

As such, we will now either completely remove these command line switches or replace them with new configuration settings. For example, a quick poll on the rsyslog mailing list did show that nobody really cared about the -l and -s options (note the absence of replies). On the other hand, we know that options like -4 (to enable IPv4 only networking) are actually being used. For such “known in-use” options we will provide alternatives via the global() configuration object.

So what’s the schedule for this? Given the fact that we assume that most folks don’t use these options, we target a quicker change cycle than for previous options. The 8.6.0 release, due for December, 2nd 2014, will emit warning messages when these options are being used – as well as telling users they need to speak up if they need that functionality in the future. Depending on the feedback we receive, the options will be removed in 8.6.1 (Jan, 13th 2015) or a later release.

2014-11-12

Some cleanup in upcoming rsyslog v8

Historically, the rsyslog source tree contains a lot of seldomly-used and exotic modules. Some of them even don’t work at the moment. I kept them inside the tree so that they could serve as a sample for folks trying a similar things. However, there has been discussion on the rsyslog mailing list that all of this clutters up rsyslog and makes it a bit hard to understand which modules are well maintained, which are not, and which actually do not work or just serve an exotic border case.

I think these concerns are valid. As a consequence, I will go through the codebase and remove what is not in actual use. I will keep contributed modules which are only occasionally maintained, but I will move them to their own directory (./contrib) so that folks more easily see this is not a project-maintained plugin. Actually, we gain clarity from this move, but we don’t loose anything: if someone decides to base some new code on the then-removed code, it’s still available in older git versions. So it can still be used as a template. Besides clarity, getting rid of the cruft also eases the work of maintaining the source tree and hopefully also releases work of distro packagers.

To get you an idea of what kind of things I will remove: there are some java programs inside the code, which were used in early versions of the testbench (around v5). They are no longer in any use at all. There is omoracle, which is orphaned for quite some while, and does not work any longer since the days of v6. There is obviously no interest in this plugin, otherwise folks would have stepped up and maintained it during the past 3 or 4 years that it does not work. There is sm_cust_bindcdr, which was done as part of a custom project. While we asked for permission to include this into the project (and got it ;)), the actual module is so specific that it is extremely unlikely someone else can use it. We just integrated it as an example. These kinds of things we will remove.

Note that this step probably also helps us in moving rsyslog as whole over to ASL 2.0, which is our long-term goal since long. Some of the things now being removed (omoracle, for example) would be problematic, as they are under GPL and we cannot contact the author any longer. This is a nice additional benefit of the cleanup.

2014-09-01

LinuxTag Presentation now online

I realized that I had forgotten to upload my LinuxTag Berlin 2014 presentation on rsyslog enhancements and writing external plugins. I have now uploaded it, so you can view it here:

2014-06-30

Upcoming new v8-stable

A new rsyslog v8-stable is coming up soon. It will not just be the next iteration of 8.2, instead it will be a new feature release based on the current 8.3 devel. So be prepared to welcome 8.4. Frequent followers may wonder why 8.4 is ready. Originally, we planned to release it after the summer break. The reason is simple: its ready to come up, albeit with a little less functionality than originally anticipated.

I was (and am) busy working on the rsyslog Windows Agent, which gets a fresh brush up of its engine. It’ll be even better (and faster) as before, but that also meant that I had less time to spent on Linux rsyslog. It turned out that I am primarily doing maintenance and bug fixing on v8-devel the past couple of weeks, just as it normally happens before a new stable branch comes up. So the code has matured. At the same time, we get very good feedback for 8.2 in general, which really makes us believe that v8 fully replaces v7. The bad news is that 8.3 is currently missing the promised non-C support for input modules. However, it’s easy to do this via the regular syslog() API, so this doesn’t look like it’s overly important. In short, this means 8.3 is ready for prime time and we won’t defer it for longer than really required. Just think about how many folks have asked about non-C actions or the ability to clear out dynafiles after an inactivity timeout.

We released 8.3.3 last week, and it is scheduled to be the last 8.3 version (if nothing really important comes up). I am still working on some rough edges, which I hopefully can smoothen within the next couple of days. If possible, I’ll move them into 8.4.0. I hope to be able to release 8.4.0 next week or the week thereafter, so we get a shiny new stable before the summer break.

Also, we will finally officially drop community development support for v7. This will probably even happen this week. As usual, that doesn’t mean v7 is put into the waste bin. I’ll continue to apply patches to it, and I expect that distros will carry if for a while. Even new v7 releases may happen from time to time. But it’s no longer a version that you can expect to receive community support on (of course, rsyslog support contract customers will also be supported on outdated versions, so relax if that is you – but that’s a different story).

I hope you are looking forward to 8.4. If you can, please also help with testing 8.3.3.

2014-04-072018-06-08

Introducing the rsyslog config builder tool

Wouldn’t it be great if we had an interactive tool that permitted it novices to build complex rsyslog configurations interactively? Without any need to understand the inner workings or even terminology? Indeed, that would not only be great, but in our opinion also remove a lot of pressure that we have on rsyslog’s documentation part.

In the light of this, we started to work on a tool called the “rsyslog configuration builder“. An initial preview goes life right now today and we invite everyone to play with it. The initial version is hopefully already useful for many cases. However, the primary intent is to gather community feedback, reactions and further suggestions.

The initial version has a restricted set of supported inputs and outputs, as well as other constructs. It works with rsyslog v7.6 and above. The tool can be used anonymously and configurations are kept during the session, with the session timeout being a couple of hours. So that should be a fair amount of time to build your config. For the future, we plan to permit saving the config when logged in into the site. That way, you can work multiple days on a single configuration.

We have many more enhancements on our mind, but first of all we would like to get your feedback. You can provide feedback any way you like, but we would be extremely happy if you post either to the rsyslog mailing list or create an issue in the rsyslog website’s github project.

2014-04-02

Moving rsyslog stable to v8…

I am happy to tell that I have finally finished the 8.2.0 rsyslog release and it is on its way to announcement, package build and so on. While v8 was basically finished since before last christmas, we had a couple of mostly nits holding the release. This is probably a lesson that we need to accept some nits instead of holding a release for so long.

With that said, there is still a nit: it is undecided how the new doc system shall be distributed. In 8.2.0, it will be a tarball inside the main tarball, something that already (and rightfully) drew some criticism. However, this time I have decided to keep on with the release rather than block it again. After all, it’s easy to fix this in 8.2.1 if we settle the issue quickly.

With v8 stable released, project policy is to officially stop support for v7. In any case, we’ll have a close look at 7.6 and will provide assistance in the next couple of weeks. After all, v8 is a considerable change, even some of the more exotic contributed output modules are not available with it. So there is a good point in keeping support for v7.6 at least until we really see there is no technical reason for keeping it.

I hope that v8 will be well-perceived … and look forward to hear both success and bug reports.

If you are interested in what are the big changes, please have a look at this slightly older blog post describing what’s new in the rsyslog v8 engine.

2014-02-03

the rsyslog v8 engine – what’s new?

I have written a small presentation on what has changed in the rsyslog v8 engine. It takes a developer’s perspective, but is most probably also of interest for administrators who would like to understand why the v8 engine scales out much better for slow outputs like ElasticSearch or databases.

For developers, it also contains the basic know-how needed to successfully (and without pain!) upgrade a pre-v8 output plugin to v8.

2014-01-13

rsyslog Ubuntu Packages: calling for contributors

Adiscon is providing Ubuntu packages for recent rsyslog versions since quite a while. We would now like to go one step further. I have created a git repository on github for all the package build source files and Andre (who creates the packages) will populate it shortly. First of all, this will enable all folks interested in building their own packages to do so based on what we use.

But secondly, and more importantly, we hope to attract contributors for creating even better packages. One of my personal goals would be to make this project the core of an “rsyslog official” PPA in the Ubuntu ecosystem. From what I see on the mailing list, forums and so on, Ubuntu is becoming an increasingly important platform for logging (maybe due to their quick support for things like Elasticsearch as well as their current decision not to use systemd journal?). Whatever the reason is, Ubuntu seems to become of the premier logging platforms and we would like to make the rsyslog experience on it as good as possible.

This requires most easy access to packages as well as them being well-maintained from an Ubuntu PoV. So if you like rsyslog and know your way around building packages and PPAs on Ubuntu, please consider joining this effort. Also, all feedback is very welcome.

Note that we will most probably start similar efforts for the other Adiscon-supported platforms shortly. But right now Ubuntu is our prime focus, given the visible increase in its userbase.

2013-12-16

How I maintain multiple rsyslog versions

Rsyslog is an enterprise-class project. Among others, this means we need to provide different versions for quite a while (enterprises don’t like to update every 6 month, and don’t do so for good reasons).

As such, we support multiple major versions. As of this writing, the versions used in practice are v5, v7 and v8 upcoming shortly. There are also v0,…,v4 versions out there, and what I write applys to them equally. If there is development going on for a version, there is a vX-devel branch. This is for those that want the new features, obviously at the expense of some enhanced chance for instability. And then there is vX-stable, which is the stable branch. Once something is named vX-stable, it NEVER receives new features, just bug fixes. There is one exception from the naming rules: the most current development version is held inside the git master branch — because that’s where people expect it. So, currently, there is no v8-devel branch, but it’s named “master” instead.

Maintaining multipl versions sounds scaring and cumbersome. Indeed, this fear came up when talking about maintaining multiple doc versions inside the rsyslog-doc project. Thanks to git, this is not the case. It’s actually dumb easy to do, once you get a little used to the workflow.

The key thing is to make changes in the oldest branch where they belong (with “oldest” being the one that the project really want’s to support). So if there is a bug that’s present in v7-stable, v7-devel and master, do NOT fix it in master. Checkout v7-stable and fix it there. The same applies to doc updates. Once you have fixed it, all you need to do is to merge the changes up. Especially for smaller changes, git does most of the hard plumbing. Occasionally, there may be some merge conflicts, but these can usually quickly solved. There are some rare exceptions, usually as the result of big refactoring. For doc, merge conflicts are always almot trivial to solve (at least have been in the past). It’s advisabile to merge up early. The longer you wait, the more work you need to do if a merge conflict occurs (because you probably don’t remember well enough what you did). The backdraw of that is that the git history becomes a little cluttered with merge entries, but that’s how it is (nothing in life is free ;)).

So in the workflow is as follows (I use the v7-stable as the “oldest” version in this workflow sample; that’s also the typical case):

git checkout v7-stable
update whatever you need to update
git commit
git checkout v7-devel
git pull . v7-stable
git checkout master
git pull . v7-devel
DONE

In esscence, you make the change once and take 30 seconds to merge it up. Merge conflicts happen seldom and are quite unlikely for doc changes. Those that happen are usually just additions to related areas, and so it takes another 30 seconds to fix them. All in all very painless.

If you check the git logs, you’ll often find occurences of the workflow above (just look for the merges, they really stand out).

Now there remains the question of multiple versions, not branches. For example, we have v7.4.0, 7.4.1, 7.4.2,… How do we handle updates to them. Well, easy: first of all when 7.4.x is out, 7.4.(x-1) is NEVER again change (why should it, that’s for the next version). So we actually do NOT do any updates to already released versions (again, think about the resulting mess). So it just boils down to being able to fetch the exact same version later. And this is extremely easy with git: just use a tag, that’s what this is meant for.

So whenever I do a release, the last thing I do after it is build and being announced is do a “git tag “, e.g. “git tag v7.4.7”. That way, I can later do “git checkout v7.4.7” and have exactly the version I look for. Note, though, that you need to do “git push –tags” in order to push the tags to your published repository.

Conclusion: using git branches and release tags makes it extremely easy to maintain a multitude of version. Without git, this would be an almost undoable task (we had a really hard time in those CVS days…).