Notes and write-ups on reliability engineering, automation, and operating Kubernetes / OpenShift at scale.
Testing AI Agents: An Eval Harness That Gives You Permission to Change
A new model drops every few weeks — cheaper, faster, supposedly better. And every time, the safe move is to do nothing. Not because the old one is better, but because you can’t prove the new one won’t quietly break something that works. The same paralysis shows up with any change to an agent: a refactor, a new tool, a prompt tweak. It works today, so don’t touch it. The way out of that is a test harness built for agents. Done well, it doesn’t just catch regressions — it gives you permission to change. ...