A Multi-Agent On-Call Responder That Doesn't Page Everyone
Fewer routine pages, no blind commands. This pack is a runnable on-call simulation: routine incidents meet a runbook, not a raw command loop. Severity gates, a command allowlist and human approval for destructive actions are built in. Critical incidents still page a person.
What's inside
- An explicit state graph — diagnose, search the runbook, execute, resolve or escalate — with no hidden reasoning loop.
- A non-negotiable severity gate: SEV1 and SEV2 always page a human after the agent has gathered context.
- Sandboxed runbook execution with an explicit command allowlist: no raw shell, no database access, no internet egress.
- A destructive-action classifier: restart, delete, drop, terminate and kill always need confirmation, even when the command is allowlisted. It is in the pack because of a near-miss described in the article: the agent tried to restart a primary database and was refused.
- A historical-incident harness that replays past incidents and checks whether the agent routes correctly.
For teams who want to auto-resolve the boring pages without giving an agent a blank shell. Free. The full simulation is on this page.
Let routine incidents meet a runbook, not a blind command loop. Replay an on-call simulation with severity gates, a command allowlist and human approval for destructive actions. Critical incidents still page a person. Includes historical incident replay.
A LangGraph-style state machine wired to PagerDuty, Datadog, and a runbook repo — sandboxed, severity-gated, and conservative by design — that auto-resolves the boring incidents and pages a human for everything else, with the exact allowlist logic that caught a near-disaster before it hit production.
Get the code:
github.com/mskrado/poc/oncall-multi-agent — MIT licensed,
pip install -e ".[dev]", python demo.py --scenario all.
On-call is the tax you pay for shipping. Most of it is 3am pages for problems a first-year engineer could fix in their sleep. This blueprint is the companion code to I Replaced My Whole On-Call Rotation With a Multi-Agent System — a runnable, dependency-free simulation of the architecture that got a real production on-call rotation to a 31% auto-resolution rate with zero bad resolutions in three months.
What's inside:
· An explicit state graph: diagnose → runbook search → execute_runbook → resolve | escalate, with no hidden reasoning loops
· A non-negotiable severity gate — SEV1/SEV2 incidents always page a human after the agent gathers context, no exceptions
· Sandboxed runbook execution with an explicit command allowlist — no raw shell, no database access, no internet egress
· A destructive-action classifier added after a real near-miss — restart / delete / drop / terminate / kill always need human confirmation, even when allowlisted
· A historical incident harness (fixtures/incidents.json) that replays past incidents and checks whether the agent routes correctly — the eval layer the author wished they'd built on day one
The near-disaster this blueprint is built around: in week two, the live agent tried to resolve a stuck database migration by chaining tool calls that amounted to restart postgresql-primary. It wasn't on the sandbox's allowlist, so the tool call simply failed and the agent escalated correctly. The reasoning trace showed exactly what it had attempted — with raw shell access, that call would have restarted a production database at 2am. That single near-miss is why the sandbox and the destructive-action classifier are treated as non-optional in this codebase.
Read the full breakdown: I Replaced My Whole On-Call Rotation With a Multi-Agent System. Here Is What I Got Right and What I Got Wrong.

Comments (0)
Join the conversation!