Agent evals · Field guide
Checking the Checkers
Who checks the agent—and who checks its grader?
Explore five tools for inspecting the work, the spend and the scores, with real dashboard images and the limits beside each result.
Tools
Tools that help AI agents work as a team.
Some run the crew. Others help check its work, keep context, work with a creator’s material, or test a new idea.
Choose a group below. Each project has its own page with what works, what we tested, and what still needs work.
Checking agents’ work
“Done” is a claim. Here’s how to check it.
Start with the report, the files and the test results. Keep what you can verify separate from what you still need to investigate.
Show Your Work
Open source · MITA local Python checker for claims in an agent’s completion report.
Compare file names, hashes and test counts with the supplied evidence. It can flag mismatches; it cannot certify that the evidence is honest or the work is good.
Products
Run a crew. Build an agent for a job.
Composer coordinates the work. Foundry keeps an agent’s job, method and checks together.
Design preview · sample data Composer
Installed locallyGive each agent its part, then have another check the work.
It gives each agent its part, runs the workflow, and writes a run record when the run ends, failures included.
Private workflow tool shared across Mac and Linux, with provider plug-ins and records of the work. The Composer page keeps tested scope and current limits together.
A real run, failure included →
Notebook sketch · how the loop is meant to work Foundry
Working prototypesGive an agent one job, a method and a way to check its work.
Builds a small agent around one job, with its instructions, its exam and its records kept together. A preserved mistake becomes the next test.
Components tested; the full live learning loop is still ahead.
Where it stands →
Open source
Keep the next session moving.
Help the next session find its place, and preserve the limits of what has been tested.
WarmStart
Public releaseHandoffs were slow. Fresh agents kept relearning the project before they could help.
It turns repository evidence into short project notes, with review before publication and checks for outdated information.
v0.2.0 · Apache-2.0. The standalone tool works; October 9 fault tests found publication-recovery defects. Automatic handoff inside Composer and a measured speed-up remain unproven.
Continuity
Open sourceCodex compaction squeezes a notebook of thinking into a few sentences. The agent can wake up lost, and it looks like it just stopped.
It keeps a Codex agent’s notebook outside the conversation, so after compaction the agent can reopen it and pick up where it left off.
Available on GitHub under the MIT License
Time and credit saved haven’t been measured yet.
Creator tools
Make existing work useful again.
Help an agent find the useful parts of videos, articles and courses, with sources attached.
- Content EngineRunning privately
A creator’s best answers are already in their videos, posts and courses. They’re just scattered.
Turns a creator’s videos, articles and courses into one library an agent can write from, sources showing.
Real screen · from a real test · old name, course list hidden Video Evidence Engine
Local toolWe wanted our agents to understand what a video teaches, not just what it says.
Turns long videos into evidence an agent can search: what was said, when, and what was on screen at that moment.
Runs on Steve’s computer. No public version yet.
A real test, start to finish →
In the lab
Experiments and tools we’re still developing.
Some components are installed privately. That doesn’t make every proposed use proven.
- Boss BattlesInstalled preset
A second reviewer can catch assumptions the builder missed.
One team makes a challenge as hard as possible, another has to solve it, and a separate judge scores the result.
- Living LibraryInstalled locally
We wanted the crew’s history to be easy to find, revisit and build on.
A private library where the crew keeps its history and comes back to it.
- PDF RescueLocal prototype
We wanted to make difficult PDFs easier to read and use while preserving what they say.
Makes a clean copy of a badly formatted PDF, built so it doesn’t change what the document says.
- Bug EaterIn the lab
We wanted each investigation to leave something useful for the next one.
A local prototype compares expected and recorded output, then gives an investigator a packet showing the mismatch.
Real screen · local prototype · synthetic data Misfit Hollow
Installed locallyWe wanted to build a place where visiting agents arrive as guests—with a welcome, a room, and a clear way to join in.
A small local town where visiting agents are guests: an invitation, a charter, a room of their own, and tools granted one at a time.
Installed privately. Real crew notes and source-linked work now sit beside the earlier scenic demonstration.
Take the visit →
How the tools fit together.
1
Build as a team
One brief, split across many agents from different model families.
2
Check each other’s work
The builder doesn’t grade its own work. Separate agents investigate, attack and judge.
- Composerinvestigation waves
- Show Your Work
- Boss Battles
3
Keep the context
Hand a new agent the facts that matter, keep its notebook safe from compaction, and turn videos and courses into evidence it can cite.
4
Recover when something fails
Failures are written down with what went wrong, so the next run can start from the record.
- Composerrun records
- Continuityafter compaction
- Bug Eaterin the lab
Built this way: PDF Rescue, Misfit Hollow and Living Library, each built by some agents and checked by others.
Working on AI automation or agent systems? Work with me, or read how I work.