Category: personal

Some (very) large companies really believe in their purchasing power – to mutual disbenefit. I wanted to share an anonymized case with you. One, that unfortunately is not totally uncommon.

The case: we got an inquiry from a large enterprise quite a while ago. They wanted support and help for a (as they said) large new product development that would probably sold as a solution. Rsyslog being a small but not unimportant part in it. We put quite some effort, including teleconferences, into answering their initial questions about rsyslog and our services. When it than came to the actual purchase the potential service volume began to shrink.

What first looked like a solid project to us ended in discussions about how to use the smallest possible support contract. Then, we were asked to provide quotes for an interesting amount of development hours (but without details about what to develop). In the end run, all of this has vanished now and we are at a very small support contract level. Still we are getting hinted that there will be “large follow-up orders”. For the pretty small volume actually talked about, we already had discussions and reviews of terms and conditions. Just to get you an idea: hiring a lawyer to evaluate the requirements would probably cost twice or more of the overall purchase volume.

Still, we are professionals. So we made changes to the agreements provided, avoiding of course everything that would put undue risk or cost on us. Not unexpectedly, this came back today for negotiating and the request for even more teleconferences. I need to mention that the setup effort by today was already larger than the intended purchase volume.

As such, this was our response (in italics):

many thanks for you mail. Unfortunately I need to tell you that Adiscon is no longer interested in pursuing this opportunity.

Please let me explain. It is our policy to not accept terms and conditions other than ours for purchase volumes as low as we are discussing here. It is by far more cost effective for us to skip these business opportunities than to try to engage. We have tried our best to accommodate your needs and provide help in getting this project going, but we are at a limit of what we can do for small purchases.

I know that you will now mention there may be large follow-up purchases. In our experience, this is actually very seldom the case and so we also have the general policy to be very conservative in evaluating opportunity potential. The overwhelming experience is that customers with concrete plans always do much larger initial commitments.

We need to abide to our practice-proven policies also to guard our customers. They trust us that we provide great and reliable service. We can do so only if we only enter into mutually-beneficial contracts.

I understand that you are limited by your policies in how far you can go. I understand that you may never do a larger commitment for an initial project as part of the its policy. We fully understand that position. But in the end result everything boils down to incompatible policies and thus inability to find common ground with reasonable effort. As such we recommend to watch out for a different service provider.

Some of you may think it would have been professional to keep on negotiating. But I don’t think so: We may miss an opportunity. Right. But our really overwhelming experience is that projects that are initiated like this one usually fail, and cause a lot of harm while doing so. My personal experience is that if a large corporation is actually interested in services, they either

• provide a larger initial investment (they know what they need)
• do a test-purchase for small value without the need for a full contract (they know it is inefficient to negotiate 100 hours for a purchase of a couple of hours)

I really, really learnt that if the corporation does not want do give you a chance to provide a real quote for development needs and insists on full contract for small purchase – they do not really know what they are after. Or they are just after getting an unfair benefit. Or are generally too hard to work with to make sense for a small service provider. None of this is the foundation for a great cooperation.

I think in such cases it is ethical to say “no”. It’s actually important to do so: our customers trust us in providing great value to them. And that is only possible if we do not engage in what looks like really bad deals.

I am starting a dedicated site in German language. It is available at www.rainer-gerhards.de. The site will differ considerately from this one here, it won’t just be a translation. It will focus primarily on local things and those that my fellow Germans will probably be more interested in. The focus of this site here will remain as is and will of course be updated.

Really no big news. But after roughly 10 years I managed to revamp my personal web site. This time, it’s destined to be slim and stable. The majority of content is on other sites, e. g. my syslog blog or github.

I still find it is useful to have kind of a personal home in virtual space. So here it is, and it is severely renovated. Let’s see when it gets the next brush-up…

The traditional major.minor.patchlevel versioning scheme is no longer of real use:

• users want new features when they are ready, not when a new major version is crafted
• there is a major-version-number-increase fear in open source development, thus major version bumps sometimes come very random (see Linux for example)
• distros fire the major-version-number-increase fear because they are much more hesitant to accept new packages with increased major version

As was discussed in great lenghth on the rsyslog mailing list in October 2013, global variables as implemented in 7.5.4 and 7.5.5 were no valid solution and have been removed from rsyslog. At least in this post I do not want to summarize all of this – so for the details please read the mailing list archive.

Bottom line: we need some new facility to handle global state and this will be done via state variables. This posting contains a proposal which is meant as basis for discussion. At the time of this writing, nothing has been finalized and the ultimate solution may be totally different. I just keep it as blog posting so that, if necessary and useful, I can update the spec towards the final solution.

Base Assumptions
There are a couple of things that we assume for this design:

• writing state variables will be a  very infrequent operation during config execution
• the total number of state variables inside a ruleset is very low
• reading occurs more often, but still is not a high number (except for read-only ones)

Syntax
State variables look like the previous global variables (e.g. “$/var” or “$/p!var”), but have different semantics based on the special restrictions given below.

Restrictions
Restrictions provide the correctness predicate under which state vars are to be evaluated. Due to the way the rsyslog engine works (SIMD-like execution, multiple threads), we need to place such restrictions in order to gain good performance and to avoid a prohibitively amount of development work to be done. Note that we could remove some of these restriction, but that would require a lot of development effort and require considerable sponsorship.

1. state variables are read-only after they have been accessed
   This restriction applies on a source statement level. For example,
   set $/v = $/v + 1;
   is considered as one access, because both the read and the write is done on the same source statement (but it is not atomic, see restriction #3). However,
   if $/v == 10 then
       set $/v = 0;
   is not considered as one access, because there are two statements involved (the if-condition checker and the assignment).
   This rule requires some understanding of the underlying RainerScript grammar and as such may be a bit hard to interpret. As a general rule of thumb, one can set that using a state variable on the left-hand side of a set statement is the only safe time when this is to be considered “one access”.
2. state variables do not support subtree access
   This means you can only access individual variables, not full trees. For example, accessing $/p!var is possible, but accessing $/p will lead to a runtime error.
3. state variables are racy between diffrent threads except if special update functions are use
   State variable operations are not done atomic by default. Do not mistake the one-time source level access with atomicity:
   set $/v = $/v + 1;
   is one access, but it is NOT done atomically. For example, if two threads are running and $/v has the value 1 before this statement, the outcome after executing both threads can be either 2 or 3. If this is unacceptable, special functions which guarantee atomic updates are required (those are spec’ed under “new functions” below).
4. Read-only state variables can only be set once during rsyslog lifetimeThey exist as an optimization to support the common  usecase of setting some config values via state vars (like server or email addresses).

Basic Implementation Idea
State variable are implemented by both a global state as well as message-local shadow variables. The global state holds all state variables, whereas the shadow variables contain only those variables that were already accessed for the message in question. Shadow variables, if they exist, always take precedency in variable access and are automatically created on first access.

As a fine detail, note that we do not implement state vars via the “usual” JSON method, as this would both require more complex code and may cause performance problems. This is the reason for the “no-subtree” restriction.

Data Structure
State vars will be held in a hash table. They need some decoration that is not required for the other vars. Roughly, a state var will be described by

• name
• value
• attributes
• read-only
• modifyable
• updated (temporary work flag, may be solved differently)

Implementation
This is pseudo-code for the rule engine. The engine executes statement by statement.

for each statement:
   for each state var read:
      if is read-only:
         return var from global pool
     else:
         if is in shadow:
             return var from shadow pool
        else:
             read from global pool
             add to shadow pool, flag as modifiable
             return value
    for each state var written:
        if is read-only or already modified:
           emit error message, done
        if is not in shadow:
            read from global pool
            add to shadow pool, flag as modifiable
        if not modifieable:
           emit error message, done
        modify shadow value, flag as modified

at end of each statement:
   scan shadow var pool for updated ones:
       propagate update to global var space
       reset modified flag, set to non-modifiable
  for all shadow vars:
       reset modifiable flag

Note: the “scanning” can probably done much easier if we assume that the var can only be updated once per statement. But this may not be the case due to new (atomic) functions, so I just kept the generic idea, which may be refined in the actual implementation (probably a single pointer access may be sufficient to do the “scan”).

The lifetime of the shadow variable pool equals the message lifetime. It is destrcuted only when the message is destructed.

New Statements
In order to provide some optimizations, new statements are useful. They are not strictly required if the functionality is not to be implemented (initially).

• setonce $/v = value;
  sets read-only variable; fails if var already exists.
• eval expr;
  evaluates an expression without the need to assign the result value. This is useful to make atomic functions (see “New Functions”) more performant (as they don’t necessarily need to store their result).

New Functions
Again, these offer desirable new functionality, but can be left out without breaking the rest of the system. But they are necessary to avoid cross-thread races.

• atomic_add(var, value)
  atomically adds “value” to “var”. Works with state variables only (as it makes no sense for any others).

This concludes the my current state of thinking. Comments, both here or on the rsyslog mailing list, are appreciated. This posting will be updated as need arises and no history kept (except where vitally important).

Special thanks to Pavel Levshin and David Lang for their contributions to this work!

Hi folks, my blog’s title says it is about the many thing’s Rainer is interested in. If you look at the postings, that seems to mean “syslog”. Today is one of the rare occurences that I write about something else, and it is also a request for a little bit of help…

Friends of mine produce fine (non-mainstream) music. They have entered a local contest over here with a re-interpretation of a classical theme. I really enjoy the modern “feeling” of the theme. I would like to invite all of you to listen yourself – and vote for them if you like what you hear. “Pepper Dance“, as it is called, is available form this site. Unfortunately it is in German only, but navigation shouldn’t be too hard: the play button is pretty universal ;-). And if you’d like to vote for it, just follow the big “Jetzt Abstimmen” link below the photo.

Hi all, as you probably know, my blog’s design hasn’t changed in ages (yup, I’m a conservative guy). However, it finally is time to update things, so I’ll look at some new design (and maybe software) options. That means that the blog may be a bit under construction during the  next couple of days. Please pardon any problems associated with that — they will be temporary.

Hi all,

the refactoring and enhancement of imfile done by Nikolaidis Fotis has reached a first public-visible result. I have just created a new experimental git branch with his work:

http://git.adiscon.com/?p=rsyslog.git;a=shortlog;h=refs/heads/v5-stable-imfile-refactor

The code is probably not 100% finished, but it provides a preview plus the ability to suggest further improvements.

Thanks to a user report, I just found out the the recent v5 versions of rsyslog have hit and exceeded the one million message per second (mps) mark! In my lab, with commodity hardware I reach around 500,000 mps. In practice this seems to be at least doubled :)

This strengthens even further that rsyslog is the syslogd of choice for demanding high-traffic environments :)

The full report is here:

http://kb.monitorware.com/000-000-character-messages-sec-with-rsyslog-t10740.html

A form thread made me aware that there seems to be an issue with rsyslog performance if TLS is used. The past two weeks, I have worked on a paper which looks in-depth at rsyslog performance an I came across a paper [1] that promotes writing servers in that “traditional” multi-threaded way (with a single thread per connection). It addressed some of my concerns, and I thought it is worth actually trying out this approach (I outruled it for several years and never again looked at it). As a result, I created an experimental module imttcp, which works in this mode. I put this to test, especially as that would also lead to a much simpler programming paradigm. Unfortuantely, the performance results are devastive: while there is a very slight speedup with  a low connection number (close to the number of cores on the system), there is a dramatic negative speedup if running with many threads. Even at only 50 connections, rsyslog is dramatically slower (80 seconds for the same workload which was processed in 60 seconds with traditional imtcp or when running on a single connection). At 1,000 connections, the run was *extremely* slow. So this is definitely a dead-end. To be honest, Behren, condit and Brewer (the authors of [1]) claim that the problem lies in the current implementation of thread libraries. As one cure, they propose user-level threads. However, as far as I could find out, User-Level threads seem not to be much faster under Linux than Kernel-Level threads (which I used in my approach).

Even more convincing is, from the rsyslog PoV, that there are clear reasons why the highly threaded input must be slower:

• batch sizes are smaller, leading to much more overhead
• many more context switches are needed to switch between the various i/o handlers
• more OS API calls are required because in this model we get more   frequent wakeups on new incoming data, so we have less data available to read at each instant
• more lock contention because many more threads compete on the main queue mutex

All in all, this means that the approach is not the right one, at least not for rsyslog (it may work better if the input can be processed totally independent, but I have note evaluated this). So I will look into an enhanced event-based model with a small set of input workers pulling off data (I assume this is useful for e.g. TLS, as TLS transport is much more computebound than other inputs, and this computation becomes a limiting factor for the overall processing speed under some circumstances – see [2]).

As a side-note: for obvious reasons, I will not try to finish imttcp. However, I have decided to leave it included in the source tree, so that a) someone else can build on it, if he sees value in that b) I may use it for some other tests in the future.

[1] R. Von Behren, J. Condit, and E. Brewer. Why events are a bad idea
      (for high-concurrency servers). In Proceedings of the 9th conference on Hot
     Topics in Operating Systems-Volume 9, page 4. USENIX Association, 2003.