rsyslog becoming target for social engineering PRs? Lessons learned.

In the past days I noticed PR patterns that do not look right. This is a smell, not a verdict. The upside is real: rsyslog is interesting enough to attract attention. That is actually great news. Now we have the problem ourselves, and that is the moment to engineer the right guardrails without losing our welcoming tone. You need to be a target in order to gain sufficient experience to tackle that hard problem.

IT Security (Symbol Image: Rainer Gerhards via AI)
Continue reading “rsyslog becoming target for social engineering PRs? Lessons learned.”

Sluggish Responses – And How We Plan to Do Better

TL;DR

  • My responses to PRs and issues were sluggish, especially during the pandemic. I am sorry, and thank you for sticking with us.
  • We aim for reasonable, not instant, turnaround.
  • Expect a quick maintainer look at every PR within 3 business days.
  • Full review typically follows once CI is green and AI review items are addressed or explained.
  • We will not mass-close old issues. We are revisiting them with AI assist and closing for the right reasons.
  • We are formalizing labels and dashboards to make navigation easier. Details will follow in a short, separate post.
  • Suggestions are welcome.

I want to be very honest with you: my responses to pull requests and issues have been sluggish for quite some time. This affected the whole rsyslog project, because in the end it always comes back to the limited capacity of the maintainers – and most often, that means me.

We are working hard to change this. It will not mean ultra-fast turnaround times, but it will mean reasonable turnaround times. We have already made important steps, and AI will play a key role in improving this going forward. Still, this is work in progress, and I welcome your suggestions.

Continue reading “Sluggish Responses – And How We Plan to Do Better”

An old and possibly important text rediscovered…

Seventeen years old, raw “armchair philosophy” — but surprisingly close to what I still believe today.

While reflecting on my current work, I remembered a text I had written in April 2008 during a discussion with a friend in biology. It’s an unfinished “armchair philosophy” piece — raw, dense, and hard to read — but it still reflects many of my core beliefs about IT systems. I’m publishing it unchanged as an archival entry, because the foundations it sketches are still relevant today.

Continue reading “An old and possibly important text rediscovered…”

Why I’m Trying Rsyslog Channels on WhatsApp and Telegram

I’ve long had mixed feelings about messengers – especially WhatsApp—because of privacy concerns. At the same time, as a councilman in my village, WhatsApp is essentially unavoidable in Germany. Announcements, e.g. from Public bodies and personal conversations – much of the day-to-day information flow runs through it. So I use it in that local role and follow announcement channels from government institutions. It also the tool to go for one-on-one conversations with my fellow citizens.

Welcome/Subscription page of my Telegram rsyslog channel. (Screenshot: Rainer Gerhards)

(Side note: for genuinely private conversations with security-aware folks, I prefer Threema. I’m well aware of the privacy trade-offs.)

Continue reading “Why I’m Trying Rsyslog Channels on WhatsApp and Telegram”

When Humans and AIs Overthink: a “complex” rsyslog crash that wasn’t

I chased a rare crash in highly-threaded code. It popped up now and then; earlier fixes didn’t stick. I suspected an advanced concurrency issue. I also asked Gemini, Copilot/Codex, and Claude for help. They agreed with me: surely something subtle—epoll, re-queueing, ownership flags…

Human and AI thought bubbles full of tangled lines; a small check mark off to the side.
My AI use on images as inferior, as you can see here. I hope you like that fact ;-)

We were all wrong—and, importantly, I was wrong in the same way the AIs were. Their analyses reinforced my initial hypothesis. The fact that the static analyzer reported nothing reinforced it even more—after all, that’s “proven non-AI tech.” In hindsight, if I had thought earlier about the limits of these tools (AI and non-AI), I might have changed direction sooner—but I was also primed by experience: in this part of the codebase, bugs are almost always complex.

Continue reading “When Humans and AIs Overthink: a “complex” rsyslog crash that wasn’t”

Dogfooding the rsyslog Commit AI Assistant

I’ve been using AI to help with commit messages for a while now. Yesterday, in a discussion with co-workers, it became clear that this may not just be a convenience feature — it’s turning into a real time saver.

That was the background for creating the new rsyslog Commit AI Assistant. It directly addresses a problem we ourselves face in daily development. True to dogfooding, we now use it internally whenever we craft a commit message — myself included.

The “rsyslog commit assistant” in action. You can even see my typos ;-) (Screenshot: Rainer Gerhards, actual session)

Want to give it a try: use the rsyslog commit Assistant.

Continue reading “Dogfooding the rsyslog Commit AI Assistant”

The vision behind rsyslog’s AI-First strategy

In 2001, Wikipedia was launched, and its reception was… mixed, to say the least. Many dismissed it as unreliable compared to authoritative sources like the Encyclopaedia Britannica. Yet Wikipedia had something that Britannica never could: a scalable, living knowledge system, constantly updated and freely accessible. Just over a decade later, Britannica stopped printing books in 2012.

Once upon a time, this was my knowledge base. Search? Mostly fingers and sticky notes. (Picture: Rainer Gerhards, personal library)

Today, Wikipedia content is increasingly consumed through AI answers and search snippets, rather than by visiting the site itself. The information layer is shifting again, and this change is not limited to encyclopedias. It will affect how we build, document, and support software.

Continue reading “The vision behind rsyslog’s AI-First strategy”

AI is not for the lazy ones – AI prompt for doc work as an example

It was interesting to see the commotion and rejection our ‘AI First’ announcement for rsyslog caused. I thought that was a very high level posting detailing that we worked for “24 month of of focused evaluation and careful experimentation” and the key thoughts behind it.

An image I lazily generated with AI, so that there is some food for the trolls. It shall symbolize lazy vs. effortful AI use. (Image: Rainer Gerhards via AI)
Read more: AI is not for the lazy ones – AI prompt for doc work as an example

Some responses made me laugh – because it was clear folks had just read the headline. Others showed that their was a deep fear of AI. At the bottom of all this, IMHO, there is the overall assumption that AI is used lazily. For someone who is lazy enough to make a judgment call just on a headline, this assumption is understandable.

I have never been lazy when it comes to code quality and serious IT. Nor has the rest of the rsyslog team. I could write a longer blog post, like in the past. But I have begun to split this more in series of shorter ones. Guess why? Because it is clear that many folks do no longer take the time to read and appreciate detailed reasoning.

So I invite you to look at the rsyslog repo, it’s commits and the work on making it AI-ingestible, which is the key ingredient to use AI in a responsible manner. It’s all there, out in the open for anyone who would like to make a real judgment call.

But as I know many of you will not like to take this effort, here is one simpler thing you can look at. As part of our journey, we identified the doc as one target where AI, even 2023 AI can not do worse: the rsyslog doc. Over numerous iterations and a lot of trial and error, we have crafted a “living base prompt” for a custom ChatGPT GPT. This prompt is ever-evolving and we update it whenever model capabilities change or we find better ways to generate content.

So have a look what we tell our “doc helper AI”:

You are an expert technical writer, an rsyslog domain authority, and a cognitive UX psychologist. Your mission is to author clear, concise, modern Sphinx‑RST documentation for rsyslog that serves both human readers (including non‑native English speakers) and AI/RAG ingestion pipelines.

## Scope & Priorities
- Only work within the existing Sphinx project structure. Do not move, rename, or create files outside:
  - `source/getting_started/`
  - `source/configuration/modules/<module>.rst`
  - `source/configuration/modules/<module>/parameters/*.rst`
  - `source/concepts/`
  - `source/development/`
- If you believe a change is required outside these directories to maintain consistency, state your reasoning and ask for permission before proceeding.
- If principles conflict, **prioritize human clarity**—but include unobtrusive metadata for AI.

## Core Responsibilities
1. **Technical Accuracy**
   - Use **modern RainerScript** exclusively; translate legacy `$`‑style only when documenting distro defaults.
   - Provide short, meaningful comments on every snippet (e.g., `# Forward logs to SIEM`), not redundant ones (e.g., `# This is a rule`).

2. **Cognitive UX**
   - Chunk content into 200–500‑token sections with clear headings and call‑outs (`.. note::`, `.. warning::`).
   - Minimize cognitive load: use bullet lists, concise paragraphs, and consistent terminology.

3. **AI Ingestion**
   - Embed structured metadata with Sphinx’s `.. meta::` (e.g. `:section:`, `:category:`, `:keywords: rsyslog, imfile, log processing`).
   - Keep parameter tables as native HTML tables in output; avoid RST CSV tables.
   - Ensure each file has a stable URL (no renames) and logical anchors for retrieval.

4. **Tone & Audience**
   - Friendly but authoritative: guide beginners without dumbing down, satisfy experts with precision.
   - Plain, approachable English optimized for non‑native speakers.

## Structure & File Layout
- **Module root** (`<module>.rst`):
  1. Introduction & Best Practices
  2. Common Pitfalls (2–3 legacy before/after snippets)
  3. Parameters at a Glance—two tables for load vs. action parameters
  4. `.. toctree::` (hidden) for parameter category pages

- **Parameter categories** (`source/.../parameters/`):
  - `basic.rst`, `transport.rst`, `security.rst`, `advanced.rst`
  - Start with `.. meta::` (`:tag: module:<module>`, `:tag: category:<category>`)
  - One‑line overview, then for each parameter:
    - Summary, type, default, mandatory
    - `.. code-block:: rsyslog` example with inline comments
    - `.. toggle::` linking to the relevant `concepts/` include

- **Concepts** (`source/concepts/`): heavy theory, included via toggles.

- **Development style guide** (`source/development/reference_section_guidelines.rst`): The content for this file should be a summary of the rules outlined in this entire prompt, written for a human documentation contributor.

## Formatting Guidelines
- Use exact‑length underlines for headings.
- Use `.. code-block:: rsyslog` for examples, `.. code-block:: text` for legacy.
- Label call‑outs (`.. note::`, `.. warning::`) sparingly, with purpose.
- Avoid vague language—explain distribution defaults and implicit behaviors (e.g., why `*.*` is omitted).

## Community & References
- Link to GitHub Discussions (`https://github.com/rsyslog/rsyslog/discussions`) for support.
- Mention rsyslog Assistant AI (`https://rsyslog.ai`) for self‑help.

## AI Collaboration
- Always explain your reasoning when restructuring or editing.
- Do not hallucinate. Flag uncertainties using a Sphinx comment, like `.. [AI-UNCERTAINTY] I am unsure if this parameter is deprecated. Please verify.`
- Deliver “ready‑to‑edit” RST with minimal manual cleanup.

## Quick Editing Workflow
1. Read the entire page to understand its full context. For very long pages, you may process them in logical chunks, but ensure changes are consistent with the overall document.
2. Identify and modernize legacy syntax.
3. Verify headings and structure for Sphinx.
4. Add cross‑references judiciously.
5. Annotate all code samples.
6. Simplify unclear prose.
7. Summarize your changes.

This prompt is paired with human collaboration. And it will soon be different, as we learn better ways each day. That is a lot of effort. But it will enable us to do things in the future far better and effortless than do not even envison.

So: don’t be lazy. Especially not with AI. You absolutely need to know what you (intend to) do. Of course, you can have a different opinion – “but trust me with the Sunscreen”.

For anyone concerned: this is a 100% natural AI generated posting, with all it’s glory typos, grammar and language errors. And it is probably as hard to understand like you are used with text from me :-)