How We Learned to Trust an AI Agent to Triage Production Incidents
How We Learned to Trust an AI Agent to Triage Production Incidents
Kiro’s model-serving team uses an agent to investigate production alarms, with access to logs and task-scoped read-only accounts. It indexes 107 incident skills and loads a playbook when an alarm calls for one. In one incident, joining failed requests to their routing records showed errors concentrated in two serving cells rather than across a model; the team added a per-cell quota check to the rate-limit playbook. A separate Sunday alarm received an evidence-linked diagnosis after about 14 minutes, leaving the decision on escalation or repair to the on-call engineer.
The learning loop had its own incident: interrupted sessions put unreviewed ticket comments into a corrections store, which later agents treated as instructions. The team added schema checks before saving lessons and pruned the store. Humans review new skills and keep authority over ticket closure and severity. The reported 250 unattended investigations in a representative month do not count human time spent reading briefs, correcting guidance or fixing root causes.
This is an operator’s account, without public ticket IDs, before-and-after skill diffs or a later trace proving that an agent read and applied the new per-cell check. A skill can be saved without being retrieved, and retrieved without producing a correct diagnosis; the proposed way to test durable learning is to link the correction, next agent’s evidence and a human-verified incident outcome.