A Multi-Agent On-Call Responder That Doesn't Page Everyone

A LangGraph-style state machine wired to PagerDuty, Datadog, and a runbook repo — sandboxed, severity-gated, and conservative by design — that auto-resolves the boring incidents and pages a human for everything else, with the exact allowlist logic that caught a near-disaster before it hit production.

Get the code: 

github.com/mskrado/poc/oncall-multi-agent — MIT licensed, 

pip install -e ".[dev]", python demo.py --scenario all.

On-call is the tax you pay for shipping. Most of it is 3am pages for problems a first-year engineer could fix in their sleep. This blueprint is the companion code to I Replaced My Whole On-Call Rotation With a Multi-Agent System — a runnable, dependency-free simulation of the architecture that got a real production on-call rotation to a 31% auto-resolution rate with zero bad resolutions in three months.

What's inside:

·        An explicit state graph: diagnose → runbook search → execute_runbook → resolve | escalate, with no hidden reasoning loops

·        A non-negotiable severity gate — SEV1/SEV2 incidents always page a human after the agent gathers context, no exceptions

·        Sandboxed runbook execution with an explicit command allowlist — no raw shell, no database access, no internet egress

·        A destructive-action classifier added after a real near-miss — restart / delete / drop / terminate / kill always need human confirmation, even when allowlisted

·        A historical incident harness (fixtures/incidents.json) that replays past incidents and checks whether the agent routes correctly — the eval layer the author wished they'd built on day one

The near-disaster this blueprint is built around: in week two, the live agent tried to resolve a stuck database migration by chaining tool calls that amounted to restart postgresql-primary. It wasn't on the sandbox's allowlist, so the tool call simply failed and the agent escalated correctly. The reasoning trace showed exactly what it had attempted — with raw shell access, that call would have restarted a production database at 2am. That single near-miss is why the sandbox and the destructive-action classifier are treated as non-optional in this codebase.

Read the full breakdown: I Replaced My Whole On-Call Rotation With a Multi-Agent System. Here Is What I Got Right and What I Got Wrong.