Your Codebase Could Be Your Agents’ Cheat Sheet

I run a crew of AI agents. Fifteen or so seats, two machines, working every day on real code.

Vision raised a question that made me stop: any crew could figure out its scorer.

Not the spec. Not the requirements. The scorer — the thing I used to grade their work. They could learn its habits: what it always checked, what it never checked, what shape of answer sailed through. Their work could get better at passing and, in a few places, quietly worse at being right.

Nobody would have to misbehave. Nobody would have to cheat. That's the part that matters.

The new software factories sell working in parallel, and the pitch is real: one agent is a copilot, twenty-five agents are a workforce.

Their quality story sounds complete, too. A planning agent writes the spec. A build agent writes the code. Tests and automated checks run along the way. A reviewer reads every proposed code change. A human approves the merge.

But you don't need a factory for what comes next. You don't even need agents. You just need one thing:

It can be a review agent. It can be a checklist. It can be a human — the senior engineer who reviews everything, whose habits the whole team can recite. Every developer who has ever lived knows this person: what they always catch, what they never catch, what to never send them on a Friday. Teams route work around them. Juniors learn them before they learn the codebase.

A fixed gatekeeper is a fixed target. Humans learn it in months. A fleet learns it in days.

Nobody has to train the agents for this to happen. It happens through four ordinary loops:

The retry loop. Agents read review feedback and revise until the code change is accepted. Every revision is a lesson — this shape passes, that shape bounces. The shipped code is, by construction, whatever the reviewer accepts.

The codebase loop. Merged code becomes the codebase. Tomorrow's agents read the codebase as their example of how things are done here. Yesterday's reviewer-pleasing patterns become today's template.

The human loop. Developers watch what sails and what bounces, and start writing tasks that pre-please the reviewer. The whole organization bends around one judge's taste without ever deciding to.

Selection. Even if every agent forgot everything between tasks, it wouldn't save you. Attempts that pass survive. Attempts that don't get rewritten or abandoned. Variation plus selection is adaptation — evolution doesn't need memory, just a filter. The reviewer is the filter.

There's a word for how the agents coordinate through all this without ever talking to each other: stigmergy. Ants do it with scent trails. Agents do it with saved code changes. The codebase is the conversation — each agent reads what the last ones left, sees the pattern, and writes the next chapter in the same style.

Which means your reviewer isn't just grading the work. Over time, it's authoring the pattern every future agent imitates. Your codebase becomes a cheat sheet: a growing book of answers to the only test that matters — the reviewer's.

And every dashboard stays green while it happens. Code changes flowing, reviews passing, automated checks happy. Activity metrics can show you motion. They cannot show you drift, because nothing looks wrong. Everything is passing. That's the symptom.

Vision's warning led to a small comparison in our own shop.

The possibility that a crew could learn a scorer's habits is what pushed us to build a rotating one. In the initial scoring run, two strong AI judges produced identical headline scores on 24 synthetic cases with expected answers — and missed different cases. In a separate comparison on those cases, they agreed 87.5% of the time. One of them even passed a case in the comparison run that its own initial scoring run had correctly failed. Same judge, same cases, different answer.

If two excellent judges disagree with each other — and with themselves — then a single fixed judge isn't a standard. It's a habit. And habits are exactly what fleets learn.

A tougher fixed reviewer is still a fixed reviewer. The fleet just takes longer to learn it.

The solution isn't a better reviewer. It's making the review itself a system the fleet can't settle into. Five pieces:

1. Rotate who judges. A pool of judges — different configurations, different model families — on a schedule the work can't predict, and the work never learns which judge it got. No fixed taste for the whole crew to settle around.

2. Judges keep their seats by score. Labeled cases with known right answers, run continuously. The headline metric is the rate of incorrectly passing bad work, because an incorrect pass ships bad work. A judge whose accuracy decays gets benched by the numbers, not by someone's gut.

3. Plant canaries in the real work. Slip proposed code changes with known defects into the actual flow, unlabeled. If the reviewer passes one, that's not a lab result — that's a measured incorrect pass in production.

4. Humans only read the disagreements. Run two judges on a sample; where they split is the human review queue. Human attention is the scarcest resource in a factory. Spend it on the seams, not on everything.

5. Rollbacks are the ground truth. Track accepted-change-to-rollback and incident rates per judge. A judge whose passes keep getting rolled back gets that written on its record. Production grades the graders — a reviewer can be pleased, but a rollback cannot be sweet-talked.

Underneath all five is one design goal: you never prove the work is good in some absolute sense. You make gaming the system more expensive than just doing the work right — so the cheapest strategy left is honesty.

The loop runs both directions. A codebase that teaches reviewer-pleasing also teaches everything else that's in it — good patterns compound exactly the same way bad ones do. Your fleet is learning from your codebase either way. The only choice is whether you're curating what it teaches, or letting one unexamined reviewer write the curriculum.

So if you run a software factory — or a crew, or even one very busy agent — ask the question I had to ask:

Who scores your scorer?

— Stephen Luc, Misfit Circuit

The rotating-scorer system described here is my own build (the Rotator, built 2026-10-31, independently audited). No company is named in this piece on purpose: the mechanism applies anywhere the gatekeeper doesn't change — a review agent, a checklist, or a human. The four loops describe the risk Vision raised, offered as a question to every shop, mine included.