The Security Patch Is Ready. Now We Wait.

I am working hard to get rsyslog in the best possible “AI age” shape. We use Daybreak Blue (as much as it permits us ;-)) and also Claude Code with Defender access. We know that these tools will NOT find especially the worst things, but they can solve a lot of issues that accumulated over time.

All OSS projects have this problem. Agentic review can find things that former static analyzers and even Coverity could not find.

That said, our main problem is the coordination effort that comes with the traditional responsible disclosure workflow. IMHO it is broken, but most still don’t see this, so it remains a security-weakening requirement.

So while I appreciate help with finding and fixing especially complex issues, it is the overall orchestration that really is our bottleneck. And not only ours, IMHO.

Let me explain what this looks like for rsyslog.

The usual process is roughly:

1. A GitHub Security Advisory (GHSA) is opened, and we coordinate with the reporter.
2. We implement and test a fix.
3. We freeze the patch, notify distributions, and wait until disclosure.

I guess the big guys have either tools or sufficient people to do parts of this manually.

Open source projects usually have, using rsyslog as an example:

• A public GitHub repository.
• Public CI.
• Public workflows.
• Public package publishing, often done by third parties (PPAs, etc.).

Nothing automated privately, and private infrastructure is also hard to pay for.

This does not play well with GHSA and similar systems. We use GHSA because it is well accepted and easily available.

For responsible disclosure, we need:

1. A private repository.
2. Private CI.
3. A private workflow.
4. A private package build structure.
5. Automation, because we do not have the people to handle all of this manually.

A private repository sounds simple. But then private CI runners cost real money, and they can be far too expensive. A GHSA private fork is no solution for us: it does not even support CI.

GHSA can be read by API. Since October 2, we can even read comments—hey! But only if they are not flagged private. Meh.

For many projects, private infrastructure may simply be too expensive or take too much time to set up.

What we do for rsyslog

We have built a process around these limitations. It works, more or less, but it is not pretty:

• We have a private repository clone, synced via an agent.
• An agent session reads the GHSA, evaluates it, and prepares communication. It uses browser automation to communicate. Perverse, TBH, and a sign of how broken the official tooling is.
• Agent sessions need local resources. We have those because Adiscon, my company, which also sells rsyslog support, pays for them.
• The required desktop agents are not very stable, so the workflow gets disrupted.
• We have set up a small set of private builders covering the most important CI parts. Full-blown CI is still not possible because we cannot mirror everything—for example, Solaris resources.

With that somewhat restricted process, we eventually get to a point where the patch is ready to go. In a normal hardening workflow, this is where I would hit “merge” and things would be fixed.

Instead, now comes freeze and wait.

We use a two-day window and go public only Tuesday through Thursday. Everything else is, IMHO, even more perverse.

“Freeze and wait” means, among other things:

• Send mail to the private distros list.
• Add the private PR to a scheduler I wrote, running on a dedicated VM. Without the scheduler, I regularly cannot be at my desk at exactly the disclosure time. More than once, I have also forgotten the date or one item or another because I was busy fixing issues.
• Create a Gmail draft for the public Openwall security list. I need to schedule it manually because the integration does not support scheduling.

At disclosure time, the scheduler moves the private PR to the public repository. Then we need to babysit it, run public CI, merge it, publish the advisory, and build public packages. Sometimes late issues are detected only now.

Basically, we can never have packages available right at disclosure time. Often, we cannot even have the merge done by then.

Some may say, “Just merge it without a PR.” But main has advanced, and full CI can find other issues. The final patch can even differ from what the distributions received.

I also strongly suspect that distributions will patch only the easy ones. I have not yet run a monitor, so that is a suspicion, not a measured result.

Compatibility is another problem

One extreme case was a reported vulnerability that resulted from a feature having fallen somewhat out of time, but still being used by some users in controlled environments.

The full fix was a monster patch. It would not apply to older versions at all because the required infrastructure was missing. That is a long-standing problem with “stable” long-term releases.

I suspect that some downstream fixes may have broken functionality. Again, I have not systematically checked that, so I cannot say how widespread it was.

The bottleneck

I appreciate help finding and fixing complex security issues. We need that. But the overall orchestration is our bottleneck, and I do not think we are alone.

We are asking open source maintainers to operate a parallel private development and release system, coordinate with multiple parties, and hit a precise disclosure time—all while the public infrastructure that normally helps them is unavailable.

This takes time away from fixing and hardening the software. Sometimes it delays getting the fix to users.

I think we need to talk about that.