Tech Enthusiast. Nature Lover. Computer Geek. rsyslog lead developer.
While I have been in Tallinn to give some lectures about syslog technology in general as well as rsyslog in specific I had the idea to use that opportunity to think about crafting rsyslog tutorials in general.
For the practical session at TALTECH IT-College I have identified a couple of typical configuration tasks. As experience shows, to carry them out successfully not only rsyslog knowledge is required but general sysadmin know-how as well. Continue reading “Tutorials for rsyslog”
Did you know? The rsyslog project offers a stable release every day! The world is changing and getting faster each time. Especially software. If a bug is fixed, you want to have the fix as soon as possible. Even more so if it is security related.
Development happens in active code. When we fix something, this is done in so-called “master branch”. If you use master branch, you are covered as soon as the fix is applied.
Traditionally there are “stable builds” which are released relatively infrequently. Sometimes many months. In rsyslog’s case, only 6 weeks. For bugfixes, someone needs to backport the fix to that “stable build”. Backporting comes with its own risks, as the code is integrated into a version it was never written for.
We have much more frequently updated versions as well. They are crafted each day and contain the current latest and greatest. Including all known fixes. These daily version are usually considered as experimental or development version. Quite honestly, this is no longer the case.
Before I continue, please consider that rsyslog is a relatively small project. What is true for it may not be true for much larger ones.
In rsyslog, we have two important policies:
What does that mean? First of all, master branch actually is a stable version. The code passed all of our checks and safeguards. We do not question the stability and continue to work on new features or bug fixes. Note that the strong position here sometimes upsets contributors. We have and had PRs who take month to ripen before they are finally sufficiently good to be merged to master.
The second thing is that daily stable builds are built from master and so are also stable.
Now let’s consider what happens when it is time to create the 6-weekly stable release. Code-wise it’s pretty simple: as master is stable, we simply take master branch and declare it as the new stable version. It’s the same version as the daily build from that day. What is different is that the doc is consolidated and we prepare files nicely for package build. Then, packages are built and tested. Except for the doc, one could also have used the daily stable build.
Think about it. What it ultimately means is that the 6-week “stable” release is just a way to avoid doing more frequent updates. But the daily build is actually as stable as the stable release.
In a world of rapidly moving development, using the daily stable build has a lot of advantages. Most importantly, one gets fixes as soon as possible. Not to mention new features.
I understand that the scheduled release may be the better option for some environments. But for most, the daily stable is actually to be preferred.
Please note: daily stable builds are currently only available for Ubuntu. With our efforts towards OpenSuse Build Service we aim to make them available for a wider number of platforms.
Today’s release of rsyslog 8.1901.0 contains a small but important feature: the ability to specify a minimum batch size. It is much-needed for some outputs, with ElasticSearch (and ClickHouse) being prime examples. While I am happy I finally implemented it, I am also a bit ashamed it took me almost three and a half year since Radu Gheorghe proposed that feature in 2015.
Quick reminder on how rsyslog batches work: we receive messages and put them into queues. From these queues, we pull so-called batches (sets of messages) and have them processed by output modules. A batch can contain a given maximum number of messages (by default and depending on case around 1024 or below). If there are that many messages inside the queue, a full batch is extracted and processed. If the queue does not contain that many, whatever it currently has is taken and forms the batch. As such a batch contain as few messages as one. Continue reading “Finally … rsyslog Minimum Batch Sizes”
As we know, rsyslog uses a version number scheme of
8.<real-version>.0
where we increment <real-version> every 6 weeks with each release. The 8 and 0 are constant (well, the 0 could change to 1 with a very important patch, but in practice we have only done this once).
While this scheme has worked pretty well since we introduced it, I often see people not understanding that there is really a big difference between 8.24 and e.g. 8.40. Looking at recent trends in software versioning, we see
With the next release, will will make more clear how old a version really is. To do so, we change the version number slightly to
8.yymm.0
where yy is the two-digit year and mm the two-digit month of the release date. We release every 6 weeks, so we will never have two releases within the same month.
So the next version will be 8.1901.0 instead of 8.41.0. To make things even more clear, rsyslog visible version output will be even more up to the point: rsyslog -v will now report “8.1901.0 (aka 2019.01)“. I am right now implementing these changes. Continue reading “rsyslog version numbering change”
The new has already spread: rsyslog 7.3 is the first version that natively supports log signatures, and does so via a newly introduced open signature provider interface. A lot of my followers quickly realized that and begun to play with it. To make sense of the provider interface, one obviously also needs a signature provider. I selected the keyless signature infrastructure (KSI), which is being engineered by the OpenKSI group. Quickly, I was asked what were the compelling reasons to provide the first signature provider for this technology.
So rather than writing many mails, I thought I blog about the reason ;) Continue reading “rsyslog’s first signature provider: why Guardtime?”
With the interest in privacy concerns currently having a “PRISM-induced high”, I wanted to elaborate a little bit about what rsyslog’s Guardtime signature provider actually transmits to the signature authority.
This is a condensed post of what the provider does, highlighting the main points. If you are really concerned, remember that everything is open source. So you are invited to read the actual signature provider source, all of which is available at the rsyslog git.
The most interesting question first: the provider does only send a top-level hash to the signature authority. No actual log record will ever be sent or otherwise disclosed. Continue reading “Which data does the Guardtime signature provider transfer to external parties?”
With RESTful APIs, like for example ElasticSearch, you need to generate JSON strings. Rsyslog will soon do this in a very easy to use way. The current method is not hard either, but often looks a bit clumsy. The new way of doing things will most probably be part of the 8.33 release.
You now can define a template as follows:
template(name="outfmt" type="list" option.jsonf="on") {
property(outname="@timestamp"
name="timereported"
dateFormat="rfc3339" format="jsonf")
property(outname="host"
name="hostname" format="jsonf")
property(outname="severity"
name="syslogseverity-text" caseConversion="upper" format="jsonf")
property(outname="facility"
name="syslogfacility-text" format="jsonf")
property(outname="syslog-tag"
name="syslogtag" format="jsonf")
property(outname="source"
name="app-name" format="jsonf")
property(outname="message"
name="msg" format="jsonf")
}
This will generate JSON. Here is a pretty-printed version of the generated output:
{
"@timestamp": "2018-03-01T01:00:00+00:00",
"host": "172.20.245.8",
"severity": "DEBUG",
"facility": "local4",
"syslog-tag": "app[1666]",
"source": "app",
"message": " this is my syslog message"
}
Note: the actual output will be compact on a single “line”, as this is most useful with RESTful APIs.
We often receive requests for Debian packages. So far, we did not package for recent Debian, as the Debian maintainer, Michael Biebl, does an excellent job. Other than us, he is a real expert on Debian policies and infrastructure.
Today, we release rsyslog 8.31. This is probably one of the biggest releases in the past couple of years. While it also offers great new functionality, what really important about it is the focus on further improved software quality.
Let’s get a bit down on it. First let’s mention some important new features: