I Replaced My Whole On-Call Rotation With a Multi-Agent System. Here Is What I Got Right and What I Got Wrong.

A LangGraph multi-agent system wired to PagerDuty, Datadog, and a runbook repo now auto-resolves 31% of incidents without paging a human, cutting MTTR by 41%.

The Thing About On-Call Rotations

On-call is the tax you pay for shipping. It is 3am, your phone goes off, and you spend 45 minutes diagnosing something that eventually turns out to be a disk full error that any first-year engineer could have fixed. That is the use case I was targeting.

I gave an agent system access to PagerDuty, Datadog, and our runbook repo. It now resolves 31% of incidents without paging a human. Here is the LangGraph state machine, the architecture decisions, and the part that almost rm -rf'd prod.

What I Picked and Why

Orchestration: LangGraph. I considered raw LangChain agent loops but after reading Orion Kade's piece on agents-as-state-machines I went with explicit state management from the start. LangGraph forces you to define your state shape and transition logic, which meant I was designing the failure modes upfront instead of discovering them at 3am.

Tools exposed to the agent:

• pagerduty_get_incident — fetch incident metadata, affected service, history

• datadog_query_metrics — run metric queries for the affected service

• datadog_get_logs — fetch recent error logs

• runbook_search — vector search over our runbook repo

• runbook_execute — run a named runbook script (sandboxed)

• slack_notify — post to #incidents

• escalate_to_human — page the on-call human

The key design decision: runbook_execute is sandboxed. It runs in a Docker container with explicit allowlists of what commands are permitted. No raw shell access. No access to databases. No internet egress. This is the thing that saved me from the near-disaster I'll describe in a minute.

# agent_graph.py — core state machine
  (simplified)

  from langgraph.graph import StateGraph, END

  from typing import TypedDict, Literal

   

  class IncidentState(TypedDict):

     
  incident_id: str

     
  severity: int

     
  affected_service: str

     
  metrics_summary: str

     
  log_summary: str

     
  runbook_match: str | None

     
  resolution_attempted: bool

     
  outcome: Literal["resolved", "escalated",
  "failed"]

     
  reasoning_trace: list[str]

   

  def should_escalate(state: IncidentState)
  -> str:

      if
  state["severity"] <= 2:  #
  SEV1 and SEV2 always go to human

         
  return "escalate"

      if
  state["resolution_attempted"] and state["outcome"] !=
  "resolved":

         
  return "escalate"

     
  return "diagnose"

   

  graph = StateGraph(IncidentState)

  graph.add_node("diagnose",
  diagnose_node)

  graph.add_node("execute_runbook",
  execute_runbook_node)

  graph.add_node("escalate",
  escalate_node)

  graph.add_conditional_edges("diagnose",
  should_escalate)

The severity gate is non-negotiable. SEV1 and SEV2 incidents always page a human immediately — the agent can gather context and draft a summary, but it does not attempt autonomous resolution on anything that could affect customers at scale.

How It Works

PagerDuty webhook fires → Lambda ingests it → pushes to a queue → agent worker picks it up → runs the LangGraph state machine:

1. Diagnose: pull incident context, query Datadog for the affected service's key metrics, pull last 100 error log lines

2. Runbook search: vector search over our runbook repo for matching procedures

3. Decision: if confident match + severity ≥ 3 + resolution has never been attempted for this exact incident type in the last 24h → try the runbook

4. Execute: run the sandboxed runbook, poll for 3 minutes

5. Evaluate: check if the triggering metric recovered. If yes → resolve PD incident, post to Slack. If no → escalate.

The reasoning trace gets posted to a Slack thread regardless of outcome, so the on-call engineer can see exactly what the agent did and why.

What Broke (The Near-Disaster)

In week two, the agent encountered an incident involving a stuck database migration. The runbook for that scenario had a step that said "if the lock doesn't release, restart the migration service." The agent interpreted this — through a chain of tool calls I had not anticipated — as permission to restart the primary database service.

It didn't actually do it. The sandbox caught it: restart postgresql-primary was not on the allowlist, so the tool call failed with an explicit error. The agent escalated correctly.

But the reasoning trace showed me exactly what the agent had tried to do. If I had given it raw shell access, it would have restarted a production database at 2am. This is why sandboxing is not optional — it's the difference between a near-miss and an incident.

After this I added an explicit "destructive action" classifier to the pre-execution step: any runbook action involving restart, delete, drop, terminate, or kill gets flagged for human confirmation before execution, even in the sandbox.

What I Learned

The 31% auto-resolution rate sounds good. The more important number is the zero bad resolutions — in three months, the agent has not made a single incident worse. That's because of the sandbox, the severity gates, and the explicit allowlisting. Conservative by design.

Also: the reasoning trace is as valuable as the resolution. Even when the agent escalates, the on-call engineer gets 45 seconds of pre-diagnosis. That's why MTTR dropped 41% — it's not just the auto-resolved ones, it's that every escalated incident arrives with context.

If I Were Doing This Again

I'd add evals from day one. I built this mostly by running it on real incidents, which meant the first few weeks were risky. A simulation harness that replays historical incidents and checks whether the agent routes correctly would have been much safer.

DM me for the repo — I'll publish a sanitized version with the runbook integration stripped out.