Tech Enthusiast. Nature Lover. Computer Geek. rsyslog lead developer.
I have written a small presentation on what has changed in the rsyslog v8 engine. It takes a developer’s perspective, but is most probably also of interest for administrators who would like to understand why the v8 engine scales out much better for slow outputs like ElasticSearch or databases.
For developers, it also contains the basic know-how needed to successfully (and without pain!) upgrade a pre-v8 output plugin to v8.
Adiscon is providing Ubuntu packages for recent rsyslog versions since quite a while. We would now like to go one step further. I have created a git repository on github for all the package build source files and Andre (who creates the packages) will populate it shortly. First of all, this will enable all folks interested in building their own packages to do so based on what we use.
But secondly, and more importantly, we hope to attract contributors for creating even better packages. One of my personal goals would be to make this project the core of an “rsyslog official” PPA in the Ubuntu ecosystem. From what I see on the mailing list, forums and so on, Ubuntu is becoming an increasingly important platform for logging (maybe due to their quick support for things like Elasticsearch as well as their current decision not to use systemd journal?). Whatever the reason is, Ubuntu seems to become of the premier logging platforms and we would like to make the rsyslog experience on it as good as possible.
This requires most easy access to packages as well as them being well-maintained from an Ubuntu PoV. So if you like rsyslog and know your way around building packages and PPAs on Ubuntu, please consider joining this effort. Also, all feedback is very welcome.
Note that we will most probably start similar efforts for the other Adiscon-supported platforms shortly. But right now Ubuntu is our prime focus, given the visible increase in its userbase.
Rsyslog is an enterprise-class project. Among others, this means we need to provide different versions for quite a while (enterprises don’t like to update every 6 month, and don’t do so for good reasons).
As such, we support multiple major versions. As of this writing, the versions used in practice are v5, v7 and v8 upcoming shortly. There are also v0,…,v4 versions out there, and what I write applys to them equally. If there is development going on for a version, there is a vX-devel branch. This is for those that want the new features, obviously at the expense of some enhanced chance for instability. And then there is vX-stable, which is the stable branch. Once something is named vX-stable, it NEVER receives new features, just bug fixes. There is one exception from the naming rules: the most current development version is held inside the git master branch — because that’s where people expect it. So, currently, there is no v8-devel branch, but it’s named “master” instead.
Maintaining multipl versions sounds scaring and cumbersome. Indeed, this fear came up when talking about maintaining multiple doc versions inside the rsyslog-doc project. Thanks to git, this is not the case. It’s actually dumb easy to do, once you get a little used to the workflow.
The key thing is to make changes in the oldest branch where they belong (with “oldest” being the one that the project really want’s to support). So if there is a bug that’s present in v7-stable, v7-devel and master, do NOT fix it in master. Checkout v7-stable and fix it there. The same applies to doc updates. Once you have fixed it, all you need to do is to merge the changes up. Especially for smaller changes, git does most of the hard plumbing. Occasionally, there may be some merge conflicts, but these can usually quickly solved. There are some rare exceptions, usually as the result of big refactoring. For doc, merge conflicts are always almot trivial to solve (at least have been in the past). It’s advisabile to merge up early. The longer you wait, the more work you need to do if a merge conflict occurs (because you probably don’t remember well enough what you did). The backdraw of that is that the git history becomes a little cluttered with merge entries, but that’s how it is (nothing in life is free ;)).
So in the workflow is as follows (I use the v7-stable as the “oldest” version in this workflow sample; that’s also the typical case):
In esscence, you make the change once and take 30 seconds to merge it up. Merge conflicts happen seldom and are quite unlikely for doc changes. Those that happen are usually just additions to related areas, and so it takes another 30 seconds to fix them. All in all very painless.
If you check the git logs, you’ll often find occurences of the workflow above (just look for the merges, they really stand out).
Now there remains the question of multiple versions, not branches. For example, we have v7.4.0, 7.4.1, 7.4.2,… How do we handle updates to them. Well, easy: first of all when 7.4.x is out, 7.4.(x-1) is NEVER again change (why should it, that’s for the next version). So we actually do NOT do any updates to already released versions (again, think about the resulting mess). So it just boils down to being able to fetch the exact same version later. And this is extremely easy with git: just use a tag, that’s what this is meant for.
So whenever I do a release, the last thing I do after it is build and being announced is do a “git tag “, e.g. “git tag v7.4.7”. That way, I can later do “git checkout v7.4.7” and have exactly the version I look for. Note, though, that you need to do “git push –tags” in order to push the tags to your published repository.
Conclusion: using git branches and release tags makes it extremely easy to maintain a multitude of version. Without git, this would be an almost undoable task (we had a really hard time in those CVS days…).
Also, we used Freshmeat for most of our public announcements around the same time (a bit later, and partly togehter with sf.net). All went extremely well, until out of the sudden they had the bright idea of a “redesign” that made the site unusable. Again, bad hit for us.
Again, I would really like to have some of the cool things. With git, it’s relatively safe, as I can move the repo around quickly, and as long as the rsyslog.com site contains the main pointer to where the current official git is, the risk is very limited. But other than that, my experience is that the short-term benefits come at the risk of severe long-term problems.
I am open to really good arguments why I am wrong. One good argument would probably be an OSS complete hosting service that is in place for 15+ yrs without any interruption in user base and breaking URLs – and that is still a hot spot. I guess sf.net mostly qualifies in that category, but as I said, we had our own story with them…
I frankly admit that I am very conservative in this regard. After all, I even post things like this on my personal blog, and not on a site directly owned by Adiscon. But, you may say, you use blogger, so don’t you take a risk here? Well, you probably have noticed that I use blogger under a domain that I have full control over. So whenever they go crazy, I can move on to some other place (not totally effortlessly, but I am in control of those things).
In some pretty long discussions, it turned out that most (if not all) users were not aware that rsyslog has an official repository on github for over six month now. So it probably is a good idea to tell the world.
While it probably is obvious, I would also like to say that I accept pull requests via github. Actually, this was always the case, even when I had no repository on github. If someone sends me a pull request, I fetch from whereever that someone’s git is located, merge it into my git and push the final result. So there is nothing special with a repository that’s on github.
Please note that we currently continue to provide rsyslog via the Adiscon git. With git, it’s irrelevat where a repository is hosted — git is not a server-based system. While I like github’s interface, I do not like to lock rsyslog into github. We almost got locked into sourceforge.net in the early days of rsyslog development and I was extremely happy that we resisted to do much more than just the CVS repository on SF when they had their really bad time. Github currently for sure is the place to be, but I like the ability to move away when the next cool thing pops up. As such, I have setup a new page that describes where the rsyslog repositories can be found currently. This page will be updated as need arises.
Please note that official repositories are peers, so all are equal (minus maybe a one-minute delta during push operations). Pick whichever you like most.
My co-worker Andre had a little time and extended the rsyslog impstats analyzer to support generating graphs. IMHO this gives you fantastic insight into how the system operates. While I know that some folks already push this data to their internal health monitoring system, the beauty of the online rsyslog impstats analyzer is that you do not need to install anything — a log file with stats is all you need to get you going. Let’s look at a quick sample. This is a page returned by the analyzer’s check phase:
Based on recent discussions on the rsyslog mailing list, we have begun to start an online tool to “analyze” impstats logs. So far, it detects “obvious” problems, but it probably is a good starting point for beginners. We plan to extend this tool.
I hope it is useful.
There have been a couple of questions when rsyslog’s next stable release (7.6) will be released.
Originally, the idea was to have this done by the end of this year, which essentially means end of November. Unfortunately, this needs to be pushed into early January 2014. The reason is code quality.
In October, we discovered a larger regression and inconsistency in regard to the new variable support in rsyslog. As it turned out, the only good choice was to seriously refactor the code handling variables and message properties. Unfortunately, such a larger refactoring also means bug potential, so I don’t feel well to release 7.6 stable with just two weeks of testing on this refactored code. Looking forward, a release around mid-December sounds doable, but this means we’ll hit the holiday break with a brand-new stable release. This is something that I definitely want to avoid. As such, the current plan is to do the release shortly after the holiday break.
In order to keep this release window, I will also avoid larger modifications to the code base. That doesn’t mean things like better support for impstats support in omelasticsearch, but it may mean that the updated imfile code will probably not be merged into v7, at least not now.
Depending on how things work out with rsyslog v8, I will do another 7.7 devel series for less intrusive enhancements or move directly to v8 only. This will be also be decided around January 2014, when we hopefully got enough feedback on v8.
This is just a blog post on where to find sample of converting modules to the v8 output module interface. Additional information will be upcoming within the next days. Stay tuned.
Please bear in mind that the v8 output module interface is not stable at this time. It will very likely change within the next weeks.
Right now, the rsyslog v8 compatibility doc has some information on the new interface. It is expected that it will remain the best source to get at least an overview of the required changes.
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=1c5a17241131129e9a1c184688500ad5b1383c28
If you browse git, the commit comments should help find more examples on a module-by-module basis.
Do you remember last October? The rsyslog mailing list got very busy with things like output load balancing, global variables, and a lot of technical details of the rsyslog engine. I don’t intend to reproduce all of this here. The interested reader may simply review the rsyslog mailing list archives, starting at October 2013.
Most of these discussions focused on the evolution of the rsyslog engine. Essentially, the v7 engine brought many new things, but was constraint due to a lot of legacy, for example the output module interface which originally (roughly 8 years ago) was designed to keep it very simple to use, especially for developers who did not know about threading (which was uncommon to know at that time). Also, the engine gained lot’s of speeds by a special SIMD-like execution mode (details in mailing list archive), but that mode was counter-productive for some of the newer features. An ultimate problem were global variables, which were very hard to implement with the v7 engine. I even proposed a complete shadow variable system to make them fit into the framework, but this was -thankfully- not well received by the community at large.
Note that I wanted to re-write at least part of the core engine for quite some while, but it sounded like a task so large that I deferred it one time after another. However, seeing all the time going into the discussion, potential work-arounds (like shadow variables) and the then still-nonoptimal state I very strongly considered if it may be time for a radical change. Better spend the time on cool new things than investing that time into some “work-around” type of implementations.
With that on my mind, I had a couple of days of drafting and some talks with my peers. It turned out that we wanted to at least give this a try and see how (and if;)) things would evolve. So I did a week of test implementation, mostly focused on a new, highly concurrent output module interface. That did work out rather well, so we finally decided to go for the full length. I started rewriting the complete core rule execution engine.
And this brings us to the lucky number: let’s welcome rsyslog v8, as the new version will be named. I thought for a short moment to stick with v7, but the changes are really, really big and quite a bit of legacy and upward compatibility has been lost. For example, all output modules need to be rewritten. So while v8 may not be a great marketing tool to get it quickly into the distros, it correctly flags that some bigger changes are to be anticipated (but as usual we were very conservative with breaking existing configurations ;)).
I did not announce this effort any sooner as until now I wasn’t sure if it would really work out. Obviously, I now think thinks look sufficiently well ;-). It’s not yet deployable, but we are getting closer.
Expect a 8.1.0 release within the next couple of days. We are working on the final finishing touches, many of which involve testing. Note that 8.1.0 will be highly experimental and really is for folks who would like to get an impression of its new capabilities … and those trying to help find bugs.
There are some restrictions on supported modules and features, and some upgrades may run into trouble with that (details will be available together with the release). Also note that the rewrite is not yet 100% complete. Some of the finer details still need to be advanced, e.g. a new, higher performance and easier to use interface for output transactions. Or some aspects of message error processing … and so on. This follows our usual pattern that in a new development branch we bring in new features gradually, until we have reached the final goal. In any case, we are now pretty confident that we will reach this goal. If all goes exceptionally well, a v8-stable may come up in March 2014 (and free support for v7 will be available for another month or two so that there is time to migrate).
To facilitate this move, git branches will change. Yesterday’s master branch will become available as v7-devel (now in place) and the master branch will hold the new v8 code. Follow this blog post and the rsyslog mailing list to be sure to see exciting new versions when they become available.
Version 7-devel will get kind of a beta status, where mostly bug fixes are applied. However, some non-bugfix changes will also happen, but only after careful consideration. The focus obviously is now on v8 and this is where we will do the “cool” stuff in the future.
Note that the average user will possibly not notice much difference between v7 and v8 initially. The new engine will be most useful for power users and installations with complex rules and/or a high log volume.
I am very excited about this new branch of development and hope you’ll also be excited as it unrolls. Also my sincere thanks to everyone who waits patiently for v7 related work. I was too busy to look a that the past few weeks and will probably be occupied with v8 for a quite a bit longer.