Today’s release of rsyslog 8.1901.0 contains a small but important feature: the ability to specify a minimum batch size. It is much-needed for some outputs, with ElasticSearch (and ClickHouse) being prime examples. While I am happy I finally implemented it, I am also a bit ashamed it took me almost three and a half year since Radu Gheorghe proposed that feature in 2015.
Quick reminder on how rsyslog batches work: we receive messages and put them into queues. From these queues, we pull so-called batches (sets of messages) and have them processed by output modules. A batch can contain a given maximum number of messages (by default and depending on case around 1024 or below). If there are that many messages inside the queue, a full batch is extracted and processed. If the queue does not contain that many, whatever it currently has is taken and forms the batch. As such a batch contain as few messages as one. Continue reading “Finally … rsyslog Minimum Batch Sizes”