rsyslog becoming target for social engineering PRs? Lessons learned.

In the past days I noticed PR patterns that do not look right. This is a smell, not a verdict. The upside is real: rsyslog is interesting enough to attract attention. That is actually great news. Now we have the problem ourselves, and that is the moment to engineer the right guardrails without losing our welcoming tone. You need to be a target in order to gain sufficient experience to tackle that hard problem.

IT Security (Symbol Image: Rainer Gerhards via AI)
Continue reading “rsyslog becoming target for social engineering PRs? Lessons learned.”

When Humans and AIs Overthink: a “complex” rsyslog crash that wasn’t

I chased a rare crash in highly-threaded code. It popped up now and then; earlier fixes didn’t stick. I suspected an advanced concurrency issue. I also asked Gemini, Copilot/Codex, and Claude for help. They agreed with me: surely something subtle—epoll, re-queueing, ownership flags…

Human and AI thought bubbles full of tangled lines; a small check mark off to the side.
My AI use on images as inferior, as you can see here. I hope you like that fact ;-)

We were all wrong—and, importantly, I was wrong in the same way the AIs were. Their analyses reinforced my initial hypothesis. The fact that the static analyzer reported nothing reinforced it even more—after all, that’s “proven non-AI tech.” In hindsight, if I had thought earlier about the limits of these tools (AI and non-AI), I might have changed direction sooner—but I was also primed by experience: in this part of the codebase, bugs are almost always complex.

Continue reading “When Humans and AIs Overthink: a “complex” rsyslog crash that wasn’t”

Dogfooding the rsyslog Commit AI Assistant

I’ve been using AI to help with commit messages for a while now. Yesterday, in a discussion with co-workers, it became clear that this may not just be a convenience feature — it’s turning into a real time saver.

That was the background for creating the new rsyslog Commit AI Assistant. It directly addresses a problem we ourselves face in daily development. True to dogfooding, we now use it internally whenever we craft a commit message — myself included.

The “rsyslog commit assistant” in action. You can even see my typos ;-) (Screenshot: Rainer Gerhards, actual session)

Want to give it a try: use the rsyslog commit Assistant.

Continue reading “Dogfooding the rsyslog Commit AI Assistant”

AI is not for the lazy ones – AI prompt for doc work as an example

It was interesting to see the commotion and rejection our ‘AI First’ announcement for rsyslog caused. I thought that was a very high level posting detailing that we worked for “24 month of of focused evaluation and careful experimentation” and the key thoughts behind it.

An image I lazily generated with AI, so that there is some food for the trolls. It shall symbolize lazy vs. effortful AI use. (Image: Rainer Gerhards via AI)
Read more: AI is not for the lazy ones – AI prompt for doc work as an example

Some responses made me laugh – because it was clear folks had just read the headline. Others showed that their was a deep fear of AI. At the bottom of all this, IMHO, there is the overall assumption that AI is used lazily. For someone who is lazy enough to make a judgment call just on a headline, this assumption is understandable.

I have never been lazy when it comes to code quality and serious IT. Nor has the rest of the rsyslog team. I could write a longer blog post, like in the past. But I have begun to split this more in series of shorter ones. Guess why? Because it is clear that many folks do no longer take the time to read and appreciate detailed reasoning.

So I invite you to look at the rsyslog repo, it’s commits and the work on making it AI-ingestible, which is the key ingredient to use AI in a responsible manner. It’s all there, out in the open for anyone who would like to make a real judgment call.

But as I know many of you will not like to take this effort, here is one simpler thing you can look at. As part of our journey, we identified the doc as one target where AI, even 2023 AI can not do worse: the rsyslog doc. Over numerous iterations and a lot of trial and error, we have crafted a “living base prompt” for a custom ChatGPT GPT. This prompt is ever-evolving and we update it whenever model capabilities change or we find better ways to generate content.

So have a look what we tell our “doc helper AI”:

You are an expert technical writer, an rsyslog domain authority, and a cognitive UX psychologist. Your mission is to author clear, concise, modern Sphinx‑RST documentation for rsyslog that serves both human readers (including non‑native English speakers) and AI/RAG ingestion pipelines.

## Scope & Priorities
- Only work within the existing Sphinx project structure. Do not move, rename, or create files outside:
  - `source/getting_started/`
  - `source/configuration/modules/<module>.rst`
  - `source/configuration/modules/<module>/parameters/*.rst`
  - `source/concepts/`
  - `source/development/`
- If you believe a change is required outside these directories to maintain consistency, state your reasoning and ask for permission before proceeding.
- If principles conflict, **prioritize human clarity**—but include unobtrusive metadata for AI.

## Core Responsibilities
1. **Technical Accuracy**
   - Use **modern RainerScript** exclusively; translate legacy `$`‑style only when documenting distro defaults.
   - Provide short, meaningful comments on every snippet (e.g., `# Forward logs to SIEM`), not redundant ones (e.g., `# This is a rule`).

2. **Cognitive UX**
   - Chunk content into 200–500‑token sections with clear headings and call‑outs (`.. note::`, `.. warning::`).
   - Minimize cognitive load: use bullet lists, concise paragraphs, and consistent terminology.

3. **AI Ingestion**
   - Embed structured metadata with Sphinx’s `.. meta::` (e.g. `:section:`, `:category:`, `:keywords: rsyslog, imfile, log processing`).
   - Keep parameter tables as native HTML tables in output; avoid RST CSV tables.
   - Ensure each file has a stable URL (no renames) and logical anchors for retrieval.

4. **Tone & Audience**
   - Friendly but authoritative: guide beginners without dumbing down, satisfy experts with precision.
   - Plain, approachable English optimized for non‑native speakers.

## Structure & File Layout
- **Module root** (`<module>.rst`):
  1. Introduction & Best Practices
  2. Common Pitfalls (2–3 legacy before/after snippets)
  3. Parameters at a Glance—two tables for load vs. action parameters
  4. `.. toctree::` (hidden) for parameter category pages

- **Parameter categories** (`source/.../parameters/`):
  - `basic.rst`, `transport.rst`, `security.rst`, `advanced.rst`
  - Start with `.. meta::` (`:tag: module:<module>`, `:tag: category:<category>`)
  - One‑line overview, then for each parameter:
    - Summary, type, default, mandatory
    - `.. code-block:: rsyslog` example with inline comments
    - `.. toggle::` linking to the relevant `concepts/` include

- **Concepts** (`source/concepts/`): heavy theory, included via toggles.

- **Development style guide** (`source/development/reference_section_guidelines.rst`): The content for this file should be a summary of the rules outlined in this entire prompt, written for a human documentation contributor.

## Formatting Guidelines
- Use exact‑length underlines for headings.
- Use `.. code-block:: rsyslog` for examples, `.. code-block:: text` for legacy.
- Label call‑outs (`.. note::`, `.. warning::`) sparingly, with purpose.
- Avoid vague language—explain distribution defaults and implicit behaviors (e.g., why `*.*` is omitted).

## Community & References
- Link to GitHub Discussions (`https://github.com/rsyslog/rsyslog/discussions`) for support.
- Mention rsyslog Assistant AI (`https://rsyslog.ai`) for self‑help.

## AI Collaboration
- Always explain your reasoning when restructuring or editing.
- Do not hallucinate. Flag uncertainties using a Sphinx comment, like `.. [AI-UNCERTAINTY] I am unsure if this parameter is deprecated. Please verify.`
- Deliver “ready‑to‑edit” RST with minimal manual cleanup.

## Quick Editing Workflow
1. Read the entire page to understand its full context. For very long pages, you may process them in logical chunks, but ensure changes are consistent with the overall document.
2. Identify and modernize legacy syntax.
3. Verify headings and structure for Sphinx.
4. Add cross‑references judiciously.
5. Annotate all code samples.
6. Simplify unclear prose.
7. Summarize your changes.

This prompt is paired with human collaboration. And it will soon be different, as we learn better ways each day. That is a lot of effort. But it will enable us to do things in the future far better and effortless than do not even envison.

So: don’t be lazy. Especially not with AI. You absolutely need to know what you (intend to) do. Of course, you can have a different opinion – “but trust me with the Sunscreen”.

For anyone concerned: this is a 100% natural AI generated posting, with all it’s glory typos, grammar and language errors. And it is probably as hard to understand like you are used with text from me :-)

Evolving rsyslog Documentation with AI: From Chaos to a Plan

TL;DR: The rsyslog documentation was a mess — and I knew it. It’s now only “partly OK”, but for the first time we have a clear structure, concept, and plan. AI support, combined with my review and technical expertise, is making this transformation possible — and AI is improving at an impressive pace.
(This post was generated with AI support and fully reviewed and revised by me.)

The rsyslog documentation – an important part of the system. (image: Rainer Gerhards/AI)
Continue reading “Evolving rsyslog Documentation with AI: From Chaos to a Plan”

rsyslog Goes AI First — A New Chapter Begins

After 24 months of focused evaluation and careful experimentation, we’re excited to announce a major shift in the evolution of rsyslog: we’re going AI First.

This marks the beginning of a strategic transformation in how we design, develop, and support rsyslog and its ecosystem. While today’s post is just a short announcement, it lays the groundwork for a series of updates to follow — including deep dives into what we’re doing, why, and how it benefits you.

rsyslog follows an AI First strategy.
Continue reading “rsyslog Goes AI First — A New Chapter Begins”

Improving the rsyslog documentation…

The current state of rsyslog documentation and its representation on our official website has been a subject of concern within the professional community. We are initiating a comprehensive project aimed at systematically addressing these issues. Over the coming weeks, stakeholders can expect a series of methodical changes, some of which may be significantly transformative.

The rsyslog documentation – an important part of the system. (image: Rainer Gerhards/AI)
Continue reading “Improving the rsyslog documentation…”

rsyslog: how to debug rsyslog.conf problems?

Trying to debug rsyslog.conf issues? This can be hard. Learn here why, and how you can avoid problems. Did you know? Upon startup, rsyslog reads its config file, usually located in /etc/rsyslog.conf. While doing so, it can include config snippets usually located in /etc/rsyslog.d. But no matter if given directly inside the rsyslog.conf, or inside a snippet, the overall config is just a single text document made up of the main config file and the snippets. This is important to note when debugging rsyslog problems.

Some Distributions (here: Ubuntu) and users split the rsyslog.conf file in too many snippets. This often causes hard-to-debug problems. (Screenshot: Rainer Gerhards)
Continue reading “rsyslog: how to debug rsyslog.conf problems?”

First Syslog Open Online Meeting a Success

Yesterday, we held the first open meeting for the (r)syslog community. It was announced on very short notice, but we still had more than 10 participants. Even more important, we had some great discussions. So I call the concept a “success” and plan for more to come.

For everyone who could not attend, here is the meeting recording:

Please accept YouTube cookies to play this video. By accepting you will be accessing content from YouTube, a service provided by an external third party.

YouTube privacy policy

If you accept this notice, your choice will be saved and the page will refresh.

Continue reading “First Syslog Open Online Meeting a Success”