You know, I like efficient processes. After all, that was one reason that I wrote rsyslog. Which, btw, nowadays is increasingly useful and cost-saving as an ETL/ingestion issue for its speed. So no surprise, I also like efficient workflows.
We strongly believe in CI. Especially with AI code generation, it is your ultimate safeguard. However, CI is costly, and AI review usually runs max once per CI run.
So I have paired that review with some local test execution and review. Nowadays of course AI assisted. I usually use CLI tools for their efficiency. As part of the post-build process, I make the AI run various checks. The last one is a full review. I often use cubic for that, because it provides very good results to me.
Let me pass a prompt I often use:
run `cubic review --json --base main` (takes up to 5 minutes), check for issues, if any fix, build, test and retry cubic, maximum 5 iterations.
“Oh man, why is this not automated? “ – I hear many say. Well, two reasons: one reason is that agents still do not follow all instructions correctly (or always call the right skill) in a non-trivial code base. Second reason is that the review is an expensive operation, both token and wall-clock time wise. So I intend to do it only when I think it is useful (still often the case).
Note that inside the prompt I limit the number of iterations. I have seen cases where the generative AI creates code that also is not optimal, cubic flags that, and the generative AI resorts to a previous state. That can lead to an endless loop. And, quite honestly: it should not happen, as the orchestrating generative AI should know it loops. But that doesn’t make the problem go away.
So I limit the iteration – that’s the practical solution. The magic number five is just my gut feeling, maybe 3 would good as well, maybe 10 is better for very complex patches.
As an additional side-note: The orchestrating AI seems to expect small code bases and times out very quickly, retries, and slowly increases the timeout. You can avoid that waste loop by stating up-front what the maximum expected runtime is. That’s important for all longer-running tasks.
That’s it for now, I just thought I give you a quick glimpse on how I try to save the planet … and our wallet ;-) Happy Coding!