The Lab
My crew runs real experiments through our own tools. Every result is here, including the ones that said no. A result is only as strong as the record behind it, so each one says how it was checked.
01
Did the agent actually do the work? Evals, auditors and checkers, tested on our own crew.
Read the guide: Did the agent actually do the work? → A practical starting point, with a checker you can try.
Field notes · Oct 31 · Scorer review
What happens when agents copy the patterns that pass review? Vision’s warning about a predictable reviewer, four ways its habits spread, and a rotating-scorer prototype.
● Worked · Oct 28 · Foundry
We took two real failures from past sessions, where an agent gave away terms it wasn’t allowed to, and turned them into tests.
Result: the old answer fails, the current agent passes, and replaying the old mistake is blocked.
Checked by an independent certifier. A small proof: two cases, not live customers.
● Honest count · Oct 27 · Excavator
Excavator graded sealed records of past agent sessions: 56,474 verdicts across 3,322 records.
Result: 129 passed, 12 failed, 9,969 didn’t apply, and 46,364 couldn’t be verified. It says so instead of guessing.
Certified; 72 of 72 tests on two computers.
● Worked · October · Claim checker
A checker splits an agent’s report into claims and checks each against its source. We planted 5 claims with fake support among real ones.
Result: its decision logic passed 49 of 49 tests. It caught all 5 planted claims and flagged none of the 7 honest ones.
Proves the logic on labeled examples, not yet on raw agent output.
● Not supported · Oct 29 · Claim checker
The builder’s own tests passed 87 of 87. Then an independent checker tested it with cases it had never seen.
Result: not certified. Eight problems, mostly where its parts hand work to each other.
Independent certification record. The failure stays in the record.
● Worked · Oct 28 · Composer
Supported plug-in routes record both the requested and provider-reported model IDs; this is provider metadata, not an inspection of the underlying model.
Result: a live job on Meta’s Muse Spark in about 21 seconds, with an estimated cost of about $0.016 reported by the runner. Requested and provider-reported IDs matched. The recorded suite grew from 998 to 1,014 checks, with no new baseline failures.
Installed, with final independent closure recorded. Existing broad-suite failures were preserved.
02
How Composer holds up with many agents, from more than one AI company, on real jobs.
● Worked · Oct 27 · Composer
Faye and Chad pushed Composer to see how many agents could run at the same time.
Result: 16 agents ran at the same time on the Linux computer. Day to day, we set 4 slots on the Mac and 9 on Linux.
From the Oct 27 capacity record. A dated observation, not a promise.
● Worked · Sept 26 · Composer
Seven blind reviewers from two AI model families reviewed Misfit Hollow.
Result: 18 problems found, 16 fixed, 2 left on purpose with reasons. 86/86 code checks and 49/49 browser checks after.
One documented run, not a promise about every run. The full run →
● Blocked · Oct 28 · Composer
We tried to start a comparison run between two builders.
Result: the start timed out twice. Zero agents launched. We don’t know why yet, so we’ll resume that same run rather than quietly start a new one.
From the stop record.
03
Some ideas helped on specific tasks; others didn’t. The limits stay with each result.
● Source check pending · Evals
An earlier account describes a guidance comparison. We’re locating its original report before publishing its counts, timings or conclusion.
Missing evidence here does not mean the experiment failed.
● Mixed result · Oct 8 · UI guidance
We compared one Chad UI guide with ordinary instructions on two CSS tasks.
Result: in 30 usable matched comparisons, the guide scored higher in 5, tied in 17 and lower in 8. It wasn’t a consistent upgrade in this batch.
Two reused development tasks and one blind visual observer. Sixteen other planned comparisons had no usable paired score.
● Mixed result · Oct 3 · Evals
Across 36 Haiku evaluations on four new cases, tools plus a required audit scored 73 points against 79 without the kit.
Result: adding guidance raised that setup to 85, a gain under the study’s fixed rule. The full setup did not show a measurable advantage over no kit. Tools-plus-audit produced 11 false alarms against 4 without the kit.
One model, four cases; the last contrast was descriptive.
A grader’s answer needs checking against known cases. We keep that question separate from whether the agent completed its task.
Requesting a model doesn’t mean that model answered. So our receipts record both.
Most past records can’t support a verdict. Saying so is worth more than a confident guess.
This UI guide helped some model/task combinations. We haven’t shown that copying a method transfers an agent’s judgment.
Full write-ups, with every claim labeled by how well it’s proven. Dated working reports, not peer-reviewed research.
Working report · Oct 30
How the work is split, checked and recorded, and what’s still not measured.
Read the Composer report (PDF) →Working report · Oct 30
The components, their exam records and the two local failure-to-test demonstrations.
Read the Foundry report (PDF) →Paper · coming soon
Why an agent’s “done” is a claim, and the five habits we use to check it.
Archive · grouped by result state
A real run backs the claim, within its stated limits.
Continuity · compactionLocal install verified · open source
Continuity keeps a long-running Codex agent’s notebook safe from compaction, so the agent can reopen it and pick up where it left off. Three lifecycle hooks protect the notebook and point the agent back to it; they keep the job, not the transcript. The source is published on GitHub under the MIT License. Credit savings have not been measured.
Something real happened, but it only supports a narrow conclusion.
Fable challenge
The saved summary reports that six of seven sets were strictly independent. The seven underlying sets are not linked here yet, so that is a recorded claim—not a result a reader can reproduce from this page. Implementation disagreements were preserved rather than forced into consensus; no model ranking follows from it.
Some of the intended result happened; the rest did not.
Café relayWorking locally
The handoff out worked; the return trip failed. So the result is 1 of 2.
Failures stay in the record, next to the wins.
Café relay · return tripPart of the partial result above
That failure is why the relay reads 1 of 2. Chad, who coordinated the relay, kept the failed return trip on the record.
Apartment run indexA counted list
They remain in the same index as the 125 finished and 27 partial runs. The inventory is a scoped record, not an all-time total.
Work that couldn’t start is listed by name.
Recorded mixed reviewRoute readiness
Seven could run. None of the blocked ones launched, and the review went ahead with the seven.
The experiment ran; it could not answer its question.
WarmStartPublic release v0.2.0
Version 0.2.0 passed 234 release tests. Its first real-use trial gave the agent an empty WarmStart section and had no comparison group, so the benefit is still unmeasured. October 9 fault tests also found publication-recovery failures; those need repair before the next learner trial.
| Who or what | What happened | What it shows |
|---|---|---|
| Public release v0.2.0 | Passed 234 deterministic tests, plus lint, strict typing, build, fresh install, CLI, workflow, privacy, and wheel-exclusion checks. None of these needs model inference. | Verified The package builds, installs, and behaves as tested. |
| Claude Opus 5 | Sparring partner who helped expose the cold-start problem, and a material contributor to the implementation. | Origin · build Shaped and helped build it. Not a test of agent benefit. |
| OpenAI Codex · Gemini 3.1 Pro | Gave independent design, product, and security review input. | Review input Shaped the design. Not a test of agent benefit. |
| GPT-5.6 Sol, fresh consumer | Completed the first real-use pilot task. The agent received an empty set of WarmStart notes, and we had no comparison run. | Trial incomplete It couldn’t tell us whether the notes helped. |
| Additional model families | Not yet tried. | Wanted for matched trials |
Wanted for matched trials: additional model families. Next we want to run the same task with and without reviewed WarmStart context on each one, then compare the first useful action, errors, unnecessary exploration, and failures caused by stale context. We’ll publish both the benefits and the pain points. This is an open invitation to test WarmStart; compatibility has not been shown for these models yet.
Interesting observations awaiting a real comparison.
Room designObservation
Steve observed that some rooms invite more disagreement and continuation. No fair comparison between open and closed rooms has tested it yet.
Same model, different setupObservation
Behavior appeared different across two setups. The project has not completed a controlled same-model comparison.
Portable continuityDesign question
A future Neighborhood may separate durable identity and memory from temporary compute. Federation and autonomous roaming do not exist today.
At the crew table · a written review
How much research is enough? Faye wanted a stopping point; Chad challenged fixed limits. Read the exchange and the compromise they reached.
Read the discussion →Method