Posted on

rsyslog’s daily stable

Did you know? The rsyslog project offers a stable release every day! The world is changing and getting faster each time. Especially software. If a bug is fixed, you want to have the fix as soon as possible. Even more so if it is security related.

Development happens in active code. When we fix something, this is done in so-called “master branch”. If you use master branch, you are covered as soon as the fix is applied.

Traditionally there are “stable builds” which are released relatively infrequently. Sometimes many months. In rsyslog’s case, only 6 weeks. For bugfixes, someone needs to backport the fix to that “stable build”. Backporting comes with its own risks, as the code is integrated into a version it was never written for.

We have much more frequently updated versions as well. They are crafted each day and contain the current latest and greatest. Including all known fixes. These daily version are usually considered as experimental or development version. Quite honestly, this is no longer the case.

Before I continue, please consider that rsyslog is a relatively small project. What is true for it may not be true for much larger ones.

In rsyslog, we have two important policies:

  • new versions never break existing configurations (except for extremely important reasons) – this means you can always update to the latest version without risking that your config blows up
  • we rely on our CI – if a change passes the testbench, it basically is good to go.To complement, we also have manual reviews of critical changes. Only things we are pretty confident in go into master branch.

What does that mean? First of all, master branch actually is a stable version. The code passed all of our checks and safeguards. We do not question the stability and continue to work on new features or bug fixes. Note that the strong position here sometimes upsets contributors. We have and had PRs who take month to ripen before they are finally sufficiently good to be merged to master.

The second thing is that daily stable builds are built from master and so are also stable.

Now let’s consider what happens when it is time to create the 6-weekly stable release. Code-wise it’s pretty simple: as master is stable, we simply take master branch and declare it as the new stable version. It’s the same version as the daily build from that day. What is different is that the doc is consolidated and we prepare files nicely for package build. Then, packages are built and tested. Except for the doc, one could also have used the daily stable build.

Think about it. What it ultimately means is that the 6-week “stable” release is just a way to avoid doing more frequent updates. But the daily build is actually as stable as the stable release.

In a world of rapidly moving development, using the daily stable build has a lot of advantages. Most importantly, one gets fixes as soon as possible. Not to mention new features.

I understand that the scheduled release may be the better option for some environments. But for most, the daily stable is actually to be preferred.

Please note: daily stable builds are currently only available for Ubuntu. With our efforts towards OpenSuse Build Service we aim to make them available for a wider number of platforms.

Posted on

Overdoing Negotiations

Some (very) large companies really believe in their purchasing power – to mutual disbenefit. I wanted to share an anonymized case with you. One, that unfortunately is not totally uncommon.

The case: we got an inquiry from a large enterprise quite a while ago. They wanted support and help for a (as they said) large new product development that would probably sold as a solution. Rsyslog being a small but not unimportant part in it. We put quite some effort, including teleconferences, into answering their initial questions about rsyslog and our services. When it than came to the actual purchase the potential service volume began to shrink.

What first looked like a solid project to us ended in discussions about how to use the smallest possible support contract. Then, we were asked to provide quotes for an interesting amount of development hours (but without details about what to develop). In the end run, all of this has vanished now and we are at a very small support contract level. Still we are getting hinted that there will be “large follow-up orders”. For the pretty small volume actually talked about, we already had discussions and reviews of terms and conditions. Just to get you an idea: hiring a lawyer to evaluate the requirements would probably cost twice or more of the overall purchase volume.

Still, we are professionals. So we made changes to the agreements provided, avoiding of course everything that would put undue risk or cost on us. Not unexpectedly, this came back today for negotiating and the request for even more teleconferences. I need to mention that the setup effort by today was already larger than the intended purchase volume.

As such, this was our response (in italics):

many thanks for you mail. Unfortunately I need to tell you that Adiscon is no longer interested in pursuing this opportunity.

Please let me explain. It is our policy to not accept terms and conditions other than ours for purchase volumes as low as we are discussing here. It is by far more cost effective for us to skip these business opportunities than to try to engage. We have tried our best to accommodate your needs and provide help in getting this project going, but we are at a limit of what we can do for small purchases.

I know that you will now mention there may be large follow-up purchases. In our experience, this is actually very seldom the case and so we also have the general policy to be very conservative in evaluating opportunity potential. The overwhelming experience is that customers with concrete plans always do much larger initial commitments.

We need to abide to our practice-proven policies also to guard our customers. They trust us that we provide great and reliable service. We can do so only if we only enter into mutually-beneficial contracts.

I understand that you are limited by your policies in how far you can go. I understand that you may never do a larger commitment for an initial project as part of the its policy. We fully understand that position. But in the end result everything boils down to incompatible policies and thus inability to find common ground with reasonable effort. As such we recommend to watch out for a different service provider.

Some of you may think it would have been professional to keep on negotiating. But I don’t think so: We may miss an opportunity. Right. But our really overwhelming experience is that projects that are initiated like this one usually fail, and cause a lot of harm while doing so. My personal experience is that if a large corporation is actually interested in services, they either

  • provide a larger initial investment (they know what they need)
  • do a test-purchase for small value without the need for a full contract (they know it is inefficient to negotiate 100 hours for a purchase of a couple of hours)

I really, really learnt that if the corporation does not want do give you a chance to provide a real quote for development needs and insists on full contract for small purchase – they do not really know what they are after. Or they are just after getting an unfair benefit. Or are generally too hard to work with to make sense for a small service provider. None of this is the foundation for a great cooperation.

I think in such cases it is ethical to say “no”. It’s actually important to do so: our customers trust us in providing great value to them. And that is only possible if we do not engage in what looks like really bad deals.

Posted on

Squash your Pull Requests!

PRs should include one commit per feature or bugfix – but not more. Especially fix-up commits are really bad and we try to automatically reject them.

rsyslog CI rejects a PR with “bad” structure

A fix-up commit is one that fixes a previous commit within the same PR.The key point is that it does not correct a current coding bug, but one that would have been introduced in the same PR. The proper thing to do is melt it together with the commit that made the mistaken. It is best to not even create the fix-up commit in the first place. Use “git commit –amend” when applying the fix.

There is a hard technical reason why fix-up commits are bad: git bisect provides an easy way to find regressions. When there are commits that do not build (or where tests fail), git bisect does not work. Continue reading “Squash your Pull Requests!”

Posted on

Finally … rsyslog Minimum Batch Sizes

Today’s release of rsyslog 8.1901.0 contains a small but important feature: the ability to specify a minimum batch size. It is much-needed for some outputs, with ElasticSearch (and ClickHouse) being prime examples. While I am happy I finally implemented it, I am also a bit ashamed it took me almost three and a half year since Radu Gheorghe proposed that feature in 2015.

Quick reminder on how rsyslog batches work: we receive messages and put them into queues. From these queues, we pull so-called batches (sets of messages) and have them processed by output modules. A batch can contain a given maximum number of messages (by default and depending on case around 1024 or below). If there are that many messages inside the queue, a full batch is extracted and processed. If the queue does not contain that many, whatever it currently has is taken and forms the batch. As such a batch contain as few messages as one. Continue reading “Finally … rsyslog Minimum Batch Sizes”

Posted on

German-Language Site going online…

I am starting a dedicated site in German language. It is available at www.rainer-gerhards.de. The site will differ considerately from this one here, it won’t just be a translation. It will focus primarily on local things and those that my fellow Germans will probably be more interested in. The focus of this site here will remain as is and will of course be updated.

Posted on

rsyslog version numbering change

As we know, rsyslog uses a version number scheme of

8.<real-version>.0

where we increment <real-version> every 6 weeks with each release. The 8 and 0 are constant (well, the 0 could change to 1 with a very important patch, but in practice we have only done this once).

While this scheme has worked pretty well since we introduced it, I often see people not understanding that there is really a big difference between 8.24 and e.g. 8.40. Looking at recent trends in software versioning, we see

  1. single-number versions, e.g. in systemd
    This is actually what we use, except that we make it look like and old-style version number by the prefix 8 and suffix 0.
  2. date-based versions, e.g. by distros (Ubuntu 18.04)

With the next release, will will make more clear how old a version really is. To do so, we change the version number slightly  to

8.yymm.0

where yy is the two-digit year and mm the two-digit month of the release date. We release every 6 weeks, so we will never have two releases within the same month.

So the next version will be 8.1901.0 instead of 8.41.0. To make things even more clear, rsyslog visible version output will be even more up to the point: rsyslog -v will now report “8.1901.0 (aka 2019.01)“. I am right now implementing these changes. Continue reading “rsyslog version numbering change”

Posted on

How we found and fixed a CVE in librelp

This is a joint blog post, from Adiscon and Semmle, about the finding and fixing of CVE-2018-1000140, a security vulnerability in librelp. This was a collaborative effort by:

  • Kevin Backhouse, Semmle, Security Researcher.
  • Rainer Gerhards, Adiscon, Founder and President.
  • Bas van Schaik, Semmle, Head of Product.

We have published this post on Rainer’s blog here and the LGTM blog.

Bas originally found the vulnerability (using lgtm.com) and Rainer fixed it. Kev developed the proof-of-concept exploit.

In this blog post, we explain the cause of the bug, which is related to a subtle gotcha in the behavior of snprintf, and how it was found by a default query on https://lgtm.com/. We also demonstrate a working exploit (in a docker container, so that you can safely download it and try it for yourself). As a bonus, we give a short tutorial on how to set up rsyslog with TLS for secure communication between the client and server. Continue reading “How we found and fixed a CVE in librelp”

Posted on

Why we use Static Code Analysis

We use static code analysis for two reasons. Both of them should probably be well-know, but discussions show that that’s not always the case. So I thought writing a small blog post makes sense.

The first reason is obvious: static analyzers help us catch code problems in early stages, and they do so without any special effort needed by test engineers. The analyzer “thinks” about many cases a human being does not think about and so can catch errors that are sometimes embarrassingly obvious – albeit you would have still overlooked them. Detecting these things early saves a lot of time. So we try to run the analyzers early and often (they are also part of our CI for that reason). Continue reading “Why we use Static Code Analysis”