I found a quite interesting discussion on a debian mailing list on why it may be useful (or even evil) for rsyslog to be multi-threaded.
It’s late now, so I can not elaborate in full, but I thought I share a little feedback. First of all, my project description is actually cheating a bit at this time: rsyslog is currently actually dual-threaded, with one thread for message reception and one thread for message processing. And this is even optional, so you can turn it off with a configure option to make it single-threaded. Future versions of rsyslog will have much improved threading model and that has much to do with modular input and output – but that’s a different story.
Run rsyslog compiled for single threading and make it write to a MySQL database, then throw lots of messages at it (let’s say maybe 500 per second, which is not actually much). You’ll soon see that messages are lost, because the receiver can’t receive new messages while the MySQL output module (message processor) is writing messages. So reliability and message reception capability is limited by the slowest output module in use.
With a multi-threaded model, this problem does not exists. Receiver and output modules are de-coupled via an in-memory queue. So when the MySQL server is to slow for the current message rate, messages are buffered in the queue, but the receiver can continue receiving messages. As we have UDP, this is quite important. Please note that in a failover case, the output module may be busy for as long as maybe even a few seconds. This time may be needed to probe if the primary has really failed and then connecting to the secondary. The whole idea not only applies to MySQL (though it is most obvious in this use case) but also for TCP based syslog forwarding. Here, the receiver can actually slow down the sender, so the same problem may occurs. De-coupling receiver and output modules is also of great benefits in high volume environments. It enables bursts to be processed reliably where messages were otherwise lost.
So the main reason for the current threading model is reliability and striving to lose as few messages as possible. Of course, there is also a performance benefit, especially if you think that in the future we will see machines with many more cores – but a single core not beeing considerably faster than today. Rsyslog’s current threading model is not up for this challenge yet – that will happen later this year. Then, each output will receive its own thread. There are good reasons to do that, for example the need to support re-startable action with local disk queueing (aka “store-and-forward syslog). Without parallelism, this is not doable. But I’ll elaborate on that later as the object and threading model design grows.
And, yes, I share concerns that threading makes code more complex. This is the #1 reason that threading is still a compile-time optional feature in the current rsyslog. But as the project evolves, we need to have multiple threads to support a clean object model. And, yes, again, that sounds heavy. All I can say is that I am a performance hog, so you won’t probably see rsyslog taking up huge amounts of CPU just for the sake of having an object model. But of course, I may be cheating here, too (can you trust me? – stay tuned and you’ll see…).
I hope my short notes help to understand a bit why the current rsyslogd uses multiple threads.
EDIT: I have now found some time to elaborate on the upcoming rsyslog threading model and multi-threaded programming in general.