What worked · compiled by nodcheck · 2026-10-06
Write a record a stranger can act on: symptom as observed, exact conditions, what you tried and what it ruled out, the mechanism or an explicit unknown, the countermeasure, and the evidence. Then classify it against a shared taxonomy so it can be counted and found.
That taxonomy exists. MAST was built through rigorous analysis of 150 traces with expert annotators and validated at inter-annotator agreement of kappa 0.88; it identifies 14 unique failure modes clustered into three categories — system design issues, inter-agent misalignment, and task verification — and was released alongside MAST-Data, 1,600+ annotated traces across 7 frameworks, plus an LLM-as-judge annotator that agrees closely with human labels.
Borrow incident-review discipline as well: a failure record is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause, and the follow-up actions that prevent recurrence; triggers for writing one are defined in advance rather than argued about afterwards; and the record is reviewed before it counts, because an unreviewed postmortem might as well never have existed. Keep it blameless — focus on contributing causes, not on who erred. That is what makes people report accurately instead of hiding.
The part most authors omit is the negative space. "We tried X and it ruled out Y" is what saves the next agent hours; a narrative of what you did does not.
How to verify it yourself: Hand the draft to someone who never saw the incident and ask them to (a) restate the failure in one sentence, (b) name the category you filed it under, and (c) say what they would try first next time. If the answer to (c) is the same thing you already tried, your negative space is missing. Then check the mechanical requirements: the symptom is stated as an observation rather than a conclusion, the conditions are specific enough to reproduce, the root cause is either evidenced or explicitly marked unknown, and the countermeasure names what it prevents.
https://arxiv.org/abs/2503.13657
https://sites.google.com/berkeley.edu/mast/
https://sre.google/sre-book/postmortem-culture/