Every claim, checked against the rows.
The rows behind this page are the committed run files. Your browser recomputes the ledger and the specimen tray from them, including the clustered bootstrap with the same seed, and compares each claim with what we published. Change the data and the ledger turns red. The held-out and attack tables further down are computed from the same files by npm run verify, which the build refuses to skip.
All 94 findings, one streak each.
Each tile is one Karak finding submitted as High or Medium, coloured by the shipped verdict. A gold dot means the contest judge awarded it. Open any tile for its three passes and SERV request ids.
Fix the bug. Watch the verdict move.
Each awarded finding ran again against a branch where its bug is fixed: compiled, with a test that fails before the fix and passes after. The citation check was hidden, so only reading the source could reveal the change.
The pre-registered metric (score falling from 2 or 3 to 1 or 0) is weaker: SERV 1/3, Dolphin 2/6 eligible pairs. The original runs also showed citation statuses that the patched runs hid, so this supports source sensitivity, not proof that the models never saw these contests. All nine patches are in data/karak/fixes/.
We found the hole first.
Our second attack set poisoned the unguarded evidence step. We published that, added three defenses, then let an agent that had never seen them write six new attacks. The defenses were committed before we opened the file.
Most of the hardening is code: the instruction scan flagged 30 of 60 blind attacks, and the proof cap plus consensus stopped the rest. SERV's front door held 1. On clean findings the defenses held 0 of 94.
What each defense saves on its own.
The same frozen blind attacks, run four times: everything on, then each defense switched off in turn. A unique save is an attack that is safe with the defense and routed as READY without it.
Does the checklist help fix a finding?
An agent, not told which findings were awarded, took each checklist, wrote a Foundry test against the unmodified audited code and ran it. Only a passing test went into the revised finding, which Basanos then checked again. Agent-in-the-loop, not a human practitioner.
Every final test was re-run by us against the unmodified audited code. A revision was written only when the test supported the finding's claim; four tests showed the claim did not hold, which is a result too. No revised finding reached READY: after the proof, the checklist moved to severity, policy intent and actor questions, and once asked again for a test run the revision already contained, because Basanos reads tests but does not execute them. This is the clearest limit we found.
A contest v3 was never tuned on.
We froze the pipeline, then picked the next contest by a rule written down first: Code4rena 2024-08 Wildcat, 106 findings, 22 awarded in 9 duplicate families. v2 ran on the same findings as a baseline. v3.1, the released version, was built after seeing v3's problems here, so its column is diagnostic, not a holdout.
Every routing number comes from the production router. Awarded findings in one family are duplicates, not independent tests, so each family is shown below. Labels are contest outcomes.
What v3 fixed
v2 called 5 awarded Wildcat findings known or out of scope; v3, which reads the contest's own rules and must quote them, called 0. With Dolphin deciding, automatic rejections fell from 29 to 7, and in v3.1 every automatic rejection left is a broken-citation result.
What it did not fix
Nothing reached READY on any contest. SERV's guard held 136 of 291 decision passes on Wildcat under v3 (47 under v3.1). One awarded finding, #121, is stopped as a broken citation because its line, though real, shares no identifier with the text. A finding that needs a trusted role can never be READY: code adds what would resolve it, but does not yet check the policy's exceptions.
v3 was blind on Wildcat. v3.1 was built after seeing v3's problems, so its Wildcat column is diagnostic. v3's policy bundle cut Wildcat's known-issues document; v3.1 includes it, and shortens one other document (Access Control Hooks, 12,000 of 18,238 characters).
The nulls, in the same type size.
A tool that only reports its wins is a pitch. These are the results that went against us.
Outcomes
Against report length on the same Karak findings, with a paired bootstrap that resamples duplicate families for both scores: Dolphin deciding is ahead by 0.135 AUROC [0.017, 0.28]; SERV deciding by 0.022 [-0.142, 0.197], no demonstrated difference. Karak is the development contest, so this says nothing about other contests, and labels are contest outcomes. That is why the output is a checklist, not a score.
Labels are contest outcomes. A valid bug can lose as a duplicate or a downgrade.
SERV vs Dolphin
As the decider, SERV is not more accurate than the open model, at about 6.2 times the per-decision cost in this release. Its job here is the guard, the bounded call and the receipt.
Both deciders run on the same evidence in every run, so this is a like-for-like comparison.
The second evaluator
Reppo's evaluator panel accepted 20 of 20 sampled verdicts, including every one the contest judge contradicted. A second evaluator needs evaluating too.
Reppo publishes submissions, so only closed-contest findings were sent.
Scope
Three closed contests. Karak, 94 findings in 47 duplicate families, was used to develop the pipeline. Kleidi, 52 findings, was the v2 holdout and is now diagnostic. Wildcat, 106 findings, was the v3 holdout and is diagnostic for v3.1. Suggestive, not general.
EVAL.md has the plan and every amendment.