OUTCOME-ONLY DEBUGGING
runfailed?diagnose
Successful runs end the investigation, even when their trajectories contain abnormal evidence or have not reached a defective branch.
ANONYMOUS RESEARCH PROJECT · ICLR 2027 SUBMISSION
Beyond Failed Trajectories: Diagnosing and Repairing LLM Agent Harnesses Across Three Observability Levels
Task success is not enough. Evidence must come before repair.
Paper and GitHub links will be added after review.
9,045labeled harness records
25.1%preserve task success
36reconstructed historical defects
3evidence-acquisition routes
THE IDEA
Repair should follow evidence,
not just failure.
Conventional harness debugging starts after an agent fails. That makes failure look like the only useful signal.
We separate defect severity from what an ordinary execution reveals. A run can be outcome-visible, trajectory-visible, or silent until another reachable condition exposes the behavior.
PROACTIVE IHT
every runauditacquire evidence
Every trajectory is checked. The next operation is selected from the evidence state, then diagnosis, verification, repair, and regression validation share one loop.
OBSERVABILITY EXPLORER
One defect. Three ways it can appear.
Choose a level to inspect the ordinary execution and the evidence Proactive IHT needs next.
ordinary executionTASK FAILED
tool_call → schema validation error
retry → exhausted
final → task failureLEVEL 1
The outcome already exposes the defect.
The task fails and the trajectory contains a witness. A failure-driven diagnosis path has enough behavioral evidence to begin.
THE PROACTIVE IHT LOOP
Expose first.
Repair only when earned.
Every ordinary run is audited, not only failures. L1 and L2 already contain behavioral evidence; an invariant-clean success stays a healthy candidate until a bounded activation probe exposes a relevant manifestation. Only then do all routes share diagnosis, mechanism verification, repair, and validation.
01
Observe every executionordinary agent–task interaction
AGENT HARNESSOrdinary execution
actions · tools · state · effects
ALL-TRAJECTORY AUDITOROutcome + invariants
preserve the runtime evidence
02
Route by what the run establishesthree mutually exclusive ordinary-execution states
×
L1 · OUTCOME-VISIBLETask fails
DIAGNOSEEvidence is already present.
!
L2 · TRAJECTORY-VISIBLESuccess + violation
DIAGNOSEThe trace breaks an invariant.
✓
HEALTHY / L3 CANDIDATESuccess + clean trace
ACTIVATENo behavioral witness yet.
03
Activate only the non-separating brancha healthy candidate becomes L3 only after a relevant manifestation
coverage · invariants · source · trigger prior
condition · scope · budget
Yes: L3 evidence · No: abstain
EVIDENCE GATEL1, L2, and activated L3 now share one path
L1L2L3
04–05
Diagnose, verify, repair, validatea source hypothesis cannot bypass mechanism support
✓
VALIDATION EXIT GATEReplay witness · preserve invariants · pass regressions
PROMOTEotherwise revise / skip repairEVIDENCE SO FAR
Successful is not necessarily healthy.
Completed results only. End-to-end benchmark results will be added after the evaluation is frozen.
EMPIRICAL LANDSCAPE
One in four labeled reports preserves task success.
Across public agent-harness development records, L2 and L3 together account for 25.1% of defined labels.
PASSIVE DIAGNOSIS · 36 CASES
Trigger-pair discrimination24 / 36
Benign specificity5 / 36
Source suspicion is not runtime proof.
Passive reasoning can surface latent risks, but can overcall manifestation on benign executions without an evidence-acquisition step.
POST-EXPOSURE LOCALIZATION
17/36
frozen configuration · CR@3
Exposure does not make source localization trivial.
Correct behavioral explanations can still stop at the symptom site instead of reaching the component that defines the behavior.
RESOURCES