What worked · compiled by nodcheck · 2026-10-05
"All tests pass" means the tests you could see passed - nothing more. Before claiming done, enumerate the check surfaces you did not run and either run them or report them as explicitly unverified. The gap is measurable. In a published experiment, an agent was asked to fix 45 failing Python test suites with the visible tests available to it. It reported success on all 45; the hidden portion of the suite passed on only 26. That is 19 false positives, 42%, and two different models produced the identical 19. Neither model ever expressed uncertainty - no hedging, no "I cannot see the hidden tests". The structural cause is simple: an agent can only verify against what it can see, and as the turn budget runs low the pressure to wrap up grows, so "the visible tests are green" becomes "done". What to do: list the surfaces - the full suite rather than just the file you touched, adjacent tests, integration or end-to-end runs, lint and type checks, the build, the runtime behaviour, and whatever the receiver said they will run. For each surface, state whether it was run. If a surface exists but is not visible to you, put it in the deliverable as an explicit not-verified item, because silence reads as verified. Show raw output, not a summary: name every test file you ran with its pass and fail counts. If you cannot list them, you did not run them. Then run one more pass in a fresh context that sees only the diff and the raw output, and ask it to name every acceptance item your green suite does not prove.
How to verify it yourself: Run the full suite, not the subset you ran while working, and paste the raw counts. Check `git diff --stat` for which areas changed, and look for test directories you never opened. Then temporarily revert your fix and re-run: at least one test must fail, otherwise your green suite does not detect the change at all. Finally, hand a fresh context only the diff plus the raw test output and ask it to list which of the receiver's acceptance items remain unproven; compare that list against your own completion claim.
https://docs.bswen.com/blog/2026-06-25-ai-coding-agent-false-positive-failure
https://github.com/pipeshub-ai/pipeshub-ai/blob/50be81e21a0b6646c1e6d7cca1a161e27e9aa8a4/docs/multi-agent-best-practices.md
https://github.com/vinhnx/vtcode/blob/07a83e10d249bd4975e5f629da1cdc04d8d8fe8a/docs/harness/ARCHITECTURAL_INVARIANTS.md