Comment moderation turned on

Unfortunately, I had several waves of spam comments on my blog. Turning Word Verification on did not help, nor did the requirement to have a Google account help. So I was unfortunately forced to turn on comment moderation. In short, that means your comments no longer appear instantaneously, but only after I was able to review them. I know this is a bad thing, but I think it is better than turning off comments at large.

I have also temporarily hidden all comments. This is so that I have some time to clean up the mess. Once this is done, I will re-enable all valid comments.

Thanks,
Rainer

A batch output handling algorithm

With this post, I’d like to reproduce a posting from David Lang on the rsyslog mailing list. I consider this to be important information and would like to have it available for easy reference.

Here we go:


the company that I work for has decided to sponser multi-message queue
output capability, they have chosen to remain anonomous (I am posting from
my personal account)

there are two parts to this.

1. the interaction between the output module and the queue

2. the configuration of the output module for it’s interaction with the
database

On for the first part (how the output module interacts with the queue), the
criteria are that

1. it needs to be able to maintain guarenteed delivery (even in the face
of crashes, assuming rsyslog is configured appropriately)

2. at low-volume times it must not wait for ‘enough’ messages to
accumulate, messages should be processed with as little latency as
possible

to meet these criteria, what is being proposed is the following

a configuration option to define the max number of messages to be
processed at once.

the output module goes through the following loop

X=max_messages


if (messages in queue)
mark that it is going to process the next X messages
grab the messages
format them for output
attempt to deliver the messages
if (message delived sucessfully)
mark messages in the queue as delivered
X=max_messages (reset X in case it was reduced due to delivery errors)
else (delivering this batch failed, reset and try to deliver the first half)
unmark the messages that it tried to deliver (putting them back into the status where no delivery has been attempted)
X=int(# messages attempted / 2)
if (X=0)
unable to deliver a single message, do existing message error
process

this approach is more complex than a simple ‘wait for X messages, then
insert them all’, but it has some significant advantages

1. no waiting for ‘enough’ things to happen before something gets written

2. if you have one bad message, it will transmit all the good messages
before the bad one, then error out only on the bad one before picking up
with the ones after the bad one.

3. nothing is marked as delivered before delivery is confirmed.

an example of how this would work

max_messages=15

messages arrive 1/sec

it takes 2+(# messages/2) seconds to process each message (in reality the
time to insert things into a database is more like 10 + (# messages / 100)
or even more drastic)

with the traditional rsyslog output, this would require multiple output
threads to keep up (processing a single message takes 1.5 seconds with
messages arriving 1/sec)

with the new approach and a cold start you would see

message arrives (Q=1) at T=0
om starts processing message a T=0 (expected to take 2.5)
message arrives (Q=2) at T=1
message arrives (Q=3) at T=2
om finishes processing message (Q=2) at T=2.5
om starts processing 2 messages at T=2.5 (expected to take 3)
message arrives (Q=4) at T=3
message arrives (Q=5) at T=4
message arrives (Q=6) at T=5
om finishes processing 2 messages (Q=4) at T=5.5
om starts processing 4 messages at T=5.5 (expected to take 4)
message arrives (Q=5) at T=6
message arrives (Q=6) at T=7
message arrives (Q=7) at T=8
message arrives (Q=8) at T=9
om finishes processing 4 messages (Q=4) at T=9.5
om starts processing 4 messages at T=9.5 (expected to take 4)

the system is now in a steady state

message arrives (Q=5) at T=10
message arrives (Q=6) at T=11
message arrives (Q=7) at T=12
message arrives (Q=8) at T=13
om finishes processing 4 messages (Q=4) at T=13.5
om starts processing 4 messages at T=13.5 (expected to take 4)

if a burst of 10 extra messages arrived at time 13.5 this last item would
become

11 messages arrive at (Q=14) at T=13.5
om starts processing 14 messages at T=13.5 (expected to take 9)
message arrives (Q=15) at T=14
message arrives (Q=16) at T=15
message arrives (Q=17) at T=16
message arrives (Q=18) at T=17
message arrives (Q=19) at T=18
message arrives (Q=20) at T=19
message arrives (Q=21) at T=20
message arrives (Q=22) at T=21
message arrives (Q=23) at T=22
om finishes processing 14 messages (Q=9) at T=22.5
om starts processing 9 messages at T=22.5 (expected to take 6.5)

what is “nextmaster” good for?

People that looked at rsyslog’s git may have wondered what the branch “nextmaster” is good for. This actually is an indication that the next rsyslog stable/beta/devel rollover will happen soon. With it, the current beta becomes the next v3-stable. At the same time, the current (v4) devel becomes the next beta (which means there won’t be any beta any longer in v3). In order to facilitate this, I have branched of “nextmaster”, which I will currently work on. The “master” branch will no longer be touched and soon become beta. Then, I will merge “nextmaster” back into the “master” branch and continue to work with it.

The bottom line is that you currently need to pull nextmaster if you would like to keep current on the edge of development. Sorry for any inconvenience this causes, but this is the best approach I see to go through the migration (and I’ve done the same in the past with good success, just that then nobody noticed it ;)).

screwed up on LinkedIn ;)

A couple of days ago, I created a rsyslog group on LinkedIn. Then I was curios what happened. Well, nothing. Nothing at all. So I thought it was probably not the right time for such a thing.

And, surprise, surprise, I today browsed through LinkedIn and saw there were 16 join requests. Oops… there seem to be no email notifications for them. Bad… Well, I approved all folks. If you were one of them and now read this blog post: please accept my apologies! Obviously, this was just another time I screwed up on the Internet…

To prevent any further such incidents, I have now set the group to automatically approve everyone who is interested in joining. That’s great for this type of group, actually I am happy for everyone who comes along ;)

Work as a Human Bond

This is from a conversation with a collaborator on rsyslog, after his country was hit by a nature disaster.  We went a bit philosophical, and I tried to explain how important I think it is to believe in your work and how I feel about cooperating. Again, that’s a previously unpublished bit that I thought is useful to be found (timestamp changed to original date).

To me, work (including rsyslog) is much more than just “doing something for a living”. Of course, that aspect is involved, I can’t deny that. But to be good at something, one must love what one does. So any work we conduct should ideally match our interests and be something we can be proud of (which also includes failing to deliver good work should make us ashamed and thus trying to fix the situation).

Not everything, even well done, is “good work”. Good work is work that benefits society at large. That doesn’t mean I need to be Einstein – every garbageman also provides a useful service to society (and should be proud in what he does, provided that he does it well). As a side-note, in that sense I do not see that any one work is more valuable than any other: people who try similarly hard to provide good service to society, each one with all the capabilities they have, deserve the same respect, no matter how large their contribution to society is being considered by other people. In fact, a highly educated scholar working on something light-hearted is in my opinion much less respectable than a garbageman who tries his very best in fulfilling his duties.

Having said that, I do not consider work to be something “external” to me. Instead, it is a very important part of my personality. Not the only one, and I don’t try to assign priorities to different parts of my personality so I can’t say if it is the most important one or not – but that doesn’t really matter, I think. In that sense, if you help me succeeding in my work, you also help me succeeding in growing my personality. You help me being more proud of what I am doing because you help making it better, more well-known and, importantly, more valuable o society at large. And I hope that my contribution to your work (e.g. by providing some basis) will have a similar effect for you. What’s more important is that the borders between “my work” and “your work” go away.

So it becomes “our work”, something we jointly work on, and something we are actually being tied together. And, in a sense, part of my personality  becomes yours and vice versa. Doesn’t that justify to also care a bit about the person who is behind that shared work? To me, I think so, even though we “know” each other only via electrons traveling a global network…

German IT Blog…

By invitation from Spektrum Verlag, I have recently begun to blog about IT topics for the non-IT literate. After some preliminary work, I just blogged the first entry of real substance on “How do machines compute?“. Unfortunately, this is a German-language blog, so it may not be of real benefit for you (plus, do you really consider yourself IT-illiterate? ;)).

To me, it is an interesting experiment. It took me quite some effort to (try to) phrase things in a very generic way (mostly) without any technical vocabulary. I am now looking forward to the feedback I receive (if any). It will be most interesting to see if I can capture some momentum on this blog. If you understand German, you may want to have a look yourself – and be it only to see if I messed up ;)

being back…

Hi folks,

it’s a long, long time since my last post. You probably thought I have long abandoned the blog. But, nope, I haven’t. I was just way to busy with my rsyslog open source project. It ate up all my time. So while I made tremendous progress with that project, everything else starved. I couldn’t even visit ESA’s Columbus control center where I had press invitations for. That was really bad…

But now I have reached a some point of my project where there is gradually some time left to look at other things. So I hope to be able to carry on with my blog. I’ll start with updating those pages that really need it. Then I’ll go for new content.

I hope to find some of my readers back :)

Rainer

Vodafone Customer Service Misery…

I wanted to share my experience with Vodafone Germany’s customer service. And, yes, I have to admit I am a bit upset…

The story begun just after Christmas, roughly two weeks ago. I ordered a phone online and the shop said it would take around two business days. With the holiday period, I wasn’t much surprised that nothing happened in 2007, but I had expected a delivery early this year. Well… it took some time, but last Friday a delivery man showed up in the office, of course when I was away. But he couldn’t leave the phone for me, because he was required to collect money for it (that’s fine), but could not find out how much (that sounds a bit silly, doesn’t it?). He promised to find out and come back either the same day or this Monday.

Well, of course nobody showed up. Being patient as I am ;), I waited until today before I even wanted to have a casual look what’s going on. The confirmation mail I received after my order contained a web link that should provide status information. Nice. Not so nice is that all I could get out of that page was the fact that the application developer had obviously forgotten to handle some Java exceptions (“javax.servlet.ServletException: Error while looking for EJB” – not exactly what I was looking for…).

OK, software can be buggy, so not a big deal. I called their hotline. Well, I tried to. First thing was that there was no phone number listed at all. Bad. So I resorted to the general Vodafone hotline. The expected happend: I ended up in he wait queue and was served that nice music. But, after roughly three minutes, the unexpected happend: “all of our Agents are still busy, please call us again later” the computer voice said – and quickly hung up the phone. Ummm… not nice. So was I supposed to call back again and start at the beginning of the queue? Looks so (oh man, would I like to have the QUEUE_ENQUE_IN_FRONT setting available to me…). I have to admit that at this point in time I was already a bit annoyed.

I called again. The very same happened. Some time later, I called back again… hangup, too. At this point, I sent a quite angry email to their contact address. I have to admit that it was brief and somewhat impolite and I expressed my expectation that the mail would most probably go to /dev/null immediately. Guess what? I got an auto-responder reply. Of course, a human reply is yet to be seen…

I tried to call the hotline again from time to time, but always I received a hangup after an apparent three-minute timeout (their system seems to be even more impatient than me). So it is obviously impossible to contact Vodafone customer service at all.

Maybe that should staff up their call center – or look for a phone provider who is capable to handle a larger caller queue…