OUTCOME-ONLY DEBUGGING
runfailed?diagnose
Successful runs end the investigation, even when their trajectories contain abnormal evidence or have not reached a defective branch.
ANONYMOUS RESEARCH PROJECT · ICLR 2027 SUBMISSION
Beyond Failed Trajectories: Diagnosing and Repairing LLM Agent Harnesses Across Three Observability Levels
Task success is not enough. Evidence must come before repair.
Paper and GitHub links will be added after review.
9,045labeled harness records
25.1%preserve task success
36reconstructed historical defects
16 / 16matched settings improve over HF
THE IDEA
Repair should follow evidence,
not just failure.
Conventional harness debugging starts after an agent fails. That makes failure look like the only useful signal.
We separate defect severity from what an ordinary execution reveals. A run can be outcome-visible, trajectory-visible, or silent until another reachable condition exposes the behavior.
PROACTIVE IHT
every runauditacquire evidence
Every trajectory is checked. The next operation is selected from the evidence state, then diagnosis, verification, repair, and regression validation share one loop.
OBSERVABILITY EXPLORER
One defect. Three ways it can appear.
Choose a level to inspect the ordinary execution and the evidence Proactive IHT needs next.
ordinary executionTASK FAILED
tool_call → schema validation error
retry → exhausted
final → task failureLEVEL 1
The outcome already exposes the defect.
The task fails and the trajectory contains a witness. A failure-driven diagnosis path has enough behavioral evidence to begin.
THE PROACTIVE IHT LOOP
Expose first.
Repair only when earned.
Every ordinary run is audited, not only failures. L1 and L2 already contain behavioral evidence; an invariant-clean success stays a healthy candidate until a bounded activation probe exposes a relevant manifestation. Only then do all routes share diagnosis, mechanism verification, repair, and validation.
01
Observe every executionordinary agent–task interaction
AGENT HARNESSOrdinary execution
actions · tools · state · effects
ALL-TRAJECTORY AUDITOROutcome + invariants
preserve the runtime evidence
02
Route by what the run establishesthree mutually exclusive ordinary-execution states
×
L1 · OUTCOME-VISIBLETask fails
DIAGNOSEEvidence is already present.
!
L2 · TRAJECTORY-VISIBLESuccess + violation
DIAGNOSEThe trace breaks an invariant.
✓
HEALTHY / L3 CANDIDATESuccess + clean trace
ACTIVATENo behavioral witness yet.
03
Activate only the non-separating brancha healthy candidate becomes L3 only after a relevant manifestation
coverage · invariants · source · trigger prior
condition · scope · budget
Yes: L3 evidence · No: abstain
EVIDENCE GATEL1, L2, and activated L3 now share one path
L1L2L3
04–05
Diagnose, verify, repair, validatea source hypothesis cannot bypass mechanism support
✓
VALIDATION EXIT GATEReplay witness · preserve invariants · pass regressions
PROMOTEotherwise revise / skip repairCOMPLETED RESULTS
Evidence improves repair.
Not every route contributes equally.
We separate held-out task utility, evidence coverage, and same-suite diagnostic analysis. The results below report only completed evaluations.
MATCHED CLOSED-LOOP EVALUATION
16 / 16
benchmark × backbone settings improve over HarnessFix
+2.0 to +5.0 pp+3.6 pp average TCR
Four public benchmarks · four task backbones · three held-out test runs per row.
GPT-5-MINI COMPARISON
+2.3pp average TCR
HF 47.1vsIHT 49.4
+10.7% offline repair tokens; the gain is incremental on top of an already strong failure-driven loop.
WHERE THE MATCHED GAINS APPEAR
Every benchmark moves forward.
Mean TCR across the four matched backbones. The blue segment is the additional gain beyond HarnessFix.
GAIA
79.5 +3.9
SWE-Bench Ver.
78.1 +3.3
AppWorld
76.8 +3.6
Terminal-Bench 2
61.8 +3.4
Rows are benchmark means; the full evaluation contains four backbone-specific rows per benchmark.
QWEN3.6-FLASH ABLATION
Repair quality and route expansion are separate levers.
H057.7HARNESSFIX67.8
ROUTE EXPANSION
L1-only routing
71.2
w/o L3 activation
72.1
Full Proactive IHT
72.4
REPAIR-QUALITY REMOVALS
w/o multi-view evidence
68.9
w/o causal verification
70.2
74%
of the full HF-to-IHT gain remains with L1-only routing. Restoring L2 adds 0.9 points and the two-candidate L3 budget adds 0.3; the orthogonal evidence and verification removals cost 3.5 and 2.1 points.
All variants use the same benchmark splits, decoding settings, outer repair budget, and validation gate. The two columns are not a sequential stack.
EMPIRICAL LANDSCAPE · 9,045 LABELS
One in four labeled reports preserves task success.
L2 and L3 together make up 25.1% of defined labels. Task success is therefore not a reliable stop condition for diagnosis.
APPWORLD EVIDENCE CHARACTERIZATION
Successful trajectories carry diagnostic signal.
90training trajectories
23L1-like
failure22L2-like
success + flag45healthy / L3
candidate
failure22L2-like
success + flag45healthy / L3
candidate
activate all 4538 / 45relevant manifestations · 84.4%
These are evidence-acquisition counts, not verified defect prevalence. The production loop uses all L1/L2 routes and a two-candidate L3 budget.
DIAGNOSIS AFTER EXPOSURE · 36 CASES
17/36frozen CR@3
23/36same-suite post-hoc CR@3
Exposure does not make source localization trivial.
The frozen diagnoser often finds the behavior site rather than the component that defines it. The post-hoc row diagnoses this attribution gap; it is not a held-out estimate.
Passive trigger pairs24 / 36Benign specificity5 / 36
RESOURCES