Checking the Checkers

An agent session leaves a record: turns, tool calls, file states, test results. A usage dashboard tells you what the session cost. Burn Detector asks the ruder question: was the spend necessary for the work the session produced?

It segments a session into intervals between progress events, prices each interval, and flags three deterministic patterns of possible waste: duplicate calls with nothing changing in between, state oscillation (a file edited back to a hash it already had), and context re-ingestion (the same material supplied again and paid for again). Every flag carries its evidence — lines, tokens, the actual calls.

The control matters as much as the flags. A productive trace — retries with changed inputs, tests flipping state as tests should — draws zero flags. A detector that flags honest work is just another noise source.

Proof state: v0.2.1, certified with findings (independent audit, 2026-10-31). Public: github.com/sleepficus/burn-detector. It implements three of the six burn signals from the method write-up; the other three are named in its README as not included, because that's where they belong until they're built.

If a fixed judge is a fixed target, the judge can't sit still. The Rotator's judge is a pool, not a personality. The configuration on duty rotates daily; the design keeps the scored agent blind to the pick. Candidate configurations earn their seat by beating the incumbent on a labeled gold set: accuracy at least as good, false-pass rate no worse, strictly better on one. A drift watch benches a judge whose accuracy decays. And where judges from different model families disagree, that disagreement is the product — it becomes the human review queue.

In the calibration records for the 24-case synthetic starter set, Spark and Wonder each got 23 of 24 cases right, with zero false passes. A lenient candidate got 16 of 24 right and falsely passed 8 of all 24 cases; the promotion check refused it. In a separate head-to-head run on those cases, Spark got 22 of 24 right and Wonder got 23 of 24. Their verdicts agreed on 21 of 24 cases — 87.5%. Those are separate run records, not interchangeable scores. The three disagreements are kept for review. The starter labels were written by the builder, with judgment calls in some cases; they are not an independently validated benchmark.

Proof state: v0.1.3, certified with findings (independent audit, 2026-10-31). Not yet public.

Judge scores usually arrive as a headline: accuracy, a false-pass rate, done. Headlines hide the failures that matter. A judge that times out on its tenth case can leave "100% accuracy" standing over a run where the tenth case was never judged at all — and was waved through by a fail-open fallback.

Judge Diagnostics reads a judge result bundle and prints the accounting instead: every attempted case, sorted by outcome — completed verdicts, abstains, errors, timeouts — with what the fallback did to each one. Accuracy is printed two ways (over completed verdicts, and over everything attempted). The false-pass rate is printed under both denominators, labeled, with the case lists beside them so anyone can recompute it. Judges that agree with each other but share a miss get named as sharing a miss. Disputed labels produce a sensitivity range, never a single point.

Here is the tool running on a synthetic bundle with a planted denominator problem:

## Findings

- **false_pass** — 2 false pass(es): outcome accepts while the expected label rejects. Cases: B19, B22.
- **headline_ambiguous_denominator** — Headline 'false_pass_rate' is unlabeled: it equals the of-all-cases rate while the expected-fail rate differs. An unlabeled denominator can understate the escape rate.

## False-pass rate — both denominators, labeled

- False passes: 2 (B19, B22)
- Expected-fail cases (expected label is fail/reject): 8 — B17, B18, B19, B20, B21, B22, B23, B24
- Of ALL cases (2/24): 8.3%
- Of EXPECTED-FAIL cases (2/8): 25%

## Headline check

| Headline field | Headline value | Computed | Status |
|---|---|---|---|
| accuracy | 0.875 | verdict_only=87.5%, attempted_case=87.5% | match |
| false_pass_rate | 0.08333333333333333 | of_all_cases=8.3%, of_expected_fail=25% | ambiguous_denominator |

Open the full synthetic report. These are selected sections of the actual output, with the wording and numbers retained.

Same two false passes. One denominator says 8.3%, the other says 25%, and the unlabeled headline picked the smaller one. That gap is the whole tool.

We also blind-tested it: fresh reviewers got either this report or the raw export plus a spreadsheet. Detection tied — both arms found every planted problem, and nobody called a dirty bundle clean. The report arm used materially fewer steps. And the test caught the report itself misleading readers twice on one framing point; that framing is fixed in v0.1.1, and the finding is in its changelog, not buried.

Proof state: v0.1.1, certified with findings (independent audit, 2026-10-31); blind test verdict: qualified proceed, as first-pass triage with the raw export one click away. Public: github.com/sleepficus/judge-diagnostics.

The public Scoring Layer candidate checks factual claims against passages in pages an agent actually captured. It records the source text and its position, or explicitly records that support was not found. Rule-based results and imported AI-judge results stay separate, with a verifiable evidence chain behind them. It does not produce a combined score. That makes it possible to inspect a flag, keep a confirmed finding, or record why a false flag was dismissed.

Proof state: public Session 2 candidate, with independent review and limits recorded in its repository; public: github.com/sleepficus/scoring-layer. No dashboard — its scorecards are text, and they read fine as text.

Hand it an agent's completion report and the evidence the work supposedly produced. It extracts the checkable claims — files, hashes, counts, success assertions — and grades each one: SUPPORTED (the evidence carries it), CONTRADICTED (the evidence says otherwise), or UNVERIFIABLE (no basis either way — listed, never hidden, never rounded up to a pass).

Its doctrine, printed in its own README: this checks receipt discipline, not truth. A metal detector, not an assay office. It beeps at claims with nothing behind them; it does not certify the ones that beep clean. Deterministic checks only — no AI judge grading the AI's homework.

Proof state: public: github.com/sleepficus/show-your-work; the shop build is at v0.1.2. No dashboard, on purpose — the verdicts are the interface.

Both dashboards shown here use the same frame, and that's deliberate. Same structure on every app: header, a bold honesty line, proof-state chips, stat tiles, exactly one signature piece (the Wheel, the interval strip), subject cards, a provenance footer. One accent color per app — ember, violet — and the verdict colors mean the same thing on every page: red asks for a human look, green marks a passing check. Accents never encode verdicts.

The point of the sameness is trust at reading speed. Learn one of these pages and you know where the honesty line lives on all of them — including the chips that say what the page isn't: synthetic data, an aggregate that is not a certification, a proof state with findings attached. The dashboards run locally and use no external assets. Their figures come from the apps’ analysis and recorded outputs. The Rotator view also freezes the daily selection; scoring happens in the command-line tool. The images on this page are still screenshots, not running dashboards.

Structure is the trust. Color is the name tag.

These tools are young, and their limits are part of the exhibit:

• The Burn Detector's sample traces and the Rotator's gold set are synthetic. They prove the mechanics — flags fire on planted burn, the gate refuses a lenient judge, calibration runs end to end. They do not prove field accuracy on real-world sessions or real judging work.
• Judge Diagnostics' fixtures are synthetic too, and its blind test used agent reviewers, which is weaker evidence than the human reviewers the test plan describes. It measures reporting utility, not customer demand.
• Rotation-prevents-gaming is unproven. It's the hypothesis the Rotator is built on, stated as a hypothesis. No adaptive comparison has run.
• Every certification here is this shop's internal discipline — builder, then an independent auditor who didn't build it — and "certified with findings" means the findings travel with the claim. None of it is a third-party endorsement.
• The dashboards report recorded history. They do not certify anything by being looked at.

That's the shelf. Composer builds; these check; the disagreements come to me.