Every major agent framework markets itself as 'reasoning-first,' but under the abstraction layers they are finite state machines — and designing explicitly for that reality produces agents that are cheaper, more debuggable, and fail in predictable rather than catastrophic ways.
|
BUILDS ON Why I Removed LangChain from Our Production Stack (V1) |
The Mistake Most Teams Make
I want to be precise about what I mean when I say frameworks are lying to you, because "lying" is a strong word and I mean it technically rather than morally.
When LangChain, LlamaIndex, CrewAI, or any comparable agent framework markets itself as enabling "autonomous reasoning agents," it is making a claim about the mental model you should use to understand the system. The claim is that the agent is reasoning — deliberating, planning, adapting based on novel understanding. This mental model is wrong, and building on a wrong mental model produces systems that fail in mysterious ways.
What the agent is actually doing is: receiving a prompt with a tool list, generating a tool call or a final response, receiving the tool output or terminating. Repeat. That is a state machine. The states are: INIT → PLANNING → TOOL_CALL → AWAITING_TOOL → (loop or) RESPONDING → DONE. The transitions are triggered by the model's output token sequence. The model does not reason across state transitions — it generates the next token sequence given the current context, which includes the accumulated state.
The V1 piece in this series documented why removing LangChain from a production stack reduced time-to-ship new agents from 9 days to 2. That outcome isn't surprising once you accept the FSM model: LangChain adds abstraction layers between you and the state machine you're actually building, and those layers add complexity without adding capability.
Why It Persists
The "reasoning agent" framing persists because it is emotionally compelling and commercially useful. An agent that "reasons" is more fundable, more marketable, and more intuitive to explain to non-engineers than an agent that "transitions between states based on a token-prediction loop." Both descriptions refer to the same system.
The problem is that the compelling framing causes teams to design for the wrong failure modes. If an agent is "reasoning," then when it fails, the diagnosis is "the reasoning failed" — which leads to prompt tweaking, model upgrades, and cargo-cult fixes. If an agent is a state machine, when it fails, the diagnosis is "which transition failed, why did the model produce output that didn't match the expected transition, and is that a prompt design failure or a tool schema failure?" The FSM mental model produces actionable diagnostics. The reasoning mental model produces expensive confusion.
Yao et al.'s ReAct framework (2023) was the paper that popularized the Thought-Action-Observation loop that underlies most agent implementations today. It's worth re-reading the original paper, because what ReAct actually describes is a structured alternation between model-generated reasoning traces (Thought) and tool invocations (Action) with tool results (Observation). The key insight of the paper was that interleaving reasoning and action improves task performance compared to either pure reasoning (chain-of-thought) or pure action (direct tool call). What the paper does not claim is that the agent is reasoning in any philosophically meaningful sense — the Thought is a prompt-induced token sequence that mimics reasoning, not reasoning itself.
What the Literature Actually Says
The Berkeley Function-Calling Leaderboard (2024) produces the most practically useful data on agent capabilities, because it evaluates the one transition that matters most in any agent loop: the model's ability to correctly select a tool, correctly format its arguments from natural-language input, and handle nested or parallel tool calls. The top models on this benchmark achieve 90-95% accuracy on single-function calls. That number drops to 60-75% on nested and parallel calls. And those numbers are measured in isolation — in a real agent loop, compounding errors across multiple tool-call steps means a 4-step task with 85% accuracy per step has an expected success rate of just 52%.
This is the mathematics of why long, autonomous agent loops fail: each state transition introduces error probability, and probabilities compound multiplicatively. Designing for this means designing for short chains, explicit checkpoints, and human escalation paths — not longer reasoning loops.
Andrej Karpathy's writing on agent architecture (2024) made a related point from a different direction: the systems that work in production are the ones that constrain the state space aggressively. An agent that can call any tool in any order has an exponential state space. An agent that can only call the tools relevant to its current phase — constrained by the FSM design — has a polynomial state space. The constraint is the reliability feature.
The Architecture That Works
Here is the complete FSM definition for a document processing agent I've deployed in production. It processes contracts, extracts structured data, and routes for human review when confidence is low:
|
from enum import Enum, auto from dataclasses import dataclass, field from typing import Optional, List
class AgentState(Enum): INIT = auto() EXTRACTING = auto() VALIDATING = auto() ROUTING = auto() ESCALATING = auto() DONE = auto() ERROR = auto()
@dataclass class AgentContext: document_id: str state: AgentState = AgentState.INIT extracted: Optional[dict] = None validation_errors: List[str] = field(default_factory=list) confidence: float = 0.0 turns: int = 0 max_turns: int = 8 # hard ceiling -- never exceed
TRANSITIONS = { AgentState.INIT: [AgentState.EXTRACTING, AgentState.ERROR], AgentState.EXTRACTING: [AgentState.VALIDATING, AgentState.ERROR], AgentState.VALIDATING: [AgentState.ROUTING, AgentState.EXTRACTING, AgentState.ERROR], AgentState.ROUTING: [AgentState.DONE, AgentState.ESCALATING], AgentState.ESCALATING: [AgentState.DONE], }
def transition(ctx: AgentContext, target: AgentState) -> AgentContext: if target not in TRANSITIONS.get(ctx.state, []): raise ValueError(f"Invalid transition: {ctx.state} -> {target}") ctx.state = target ctx.turns += 1 if ctx.turns >= ctx.max_turns and ctx.state not in (AgentState.DONE, AgentState.ERROR): ctx.state = AgentState.ESCALATING # force resolution return ctx |
Three design decisions in this code are worth naming explicitly:
Hard turn ceiling: max_turns = 8 is a hard stop. No production agent should be able to run indefinitely. Every runaway agent I've seen had no ceiling. The ceiling forces the architect to think about what "the agent didn't finish in N steps" should mean — usually, human escalation.
Explicit transition table: TRANSITIONS is a whitelist, not a blacklist. The agent can only move to states defined in the table. An unexpected model output that doesn't match a valid transition raises an exception rather than silently putting the agent in an undefined state. Silent undefined states are the root cause of most agent behavior that practitioners describe as "hallucinating" — it's usually not hallucination, it's an undetected state transition failure.
State-aware context: The AgentContext carries enough information to reconstruct exactly what the agent was doing when it failed. This is the diagnostic layer. Without it, you're looking at log files trying to infer state from tool call sequences.
The corresponding system diagram:
|
DOCUMENT PROCESSING AGENT -- STATE TRANSITIONS
[INIT] ──────────────────────────────────► [ERROR] │ ▲ ▼ │ [EXTRACTING] ──── extraction fails ───────────►│ │ │ ▼ │ [VALIDATING] ──── validation fails ─► [EXTRACTING] (retry<=2) │ │ │ └──── retry limit ──► [ERROR] ▼ [ROUTING] │ │ ▼ ▼ [DONE] [ESCALATING] ──► human queue ──► [DONE] |
The retry-from-VALIDATING-to-EXTRACTING loop has an explicit limit of 2. The system counts retries in AgentContext and transitions to ERROR rather than looping forever. This is the single most important control in the design — infinite retry loops are how agents consume $400 of inference budget on a $2 task.
Failure Modes
Ghost state: The model returns output that doesn't match any valid next state. Most frameworks handle this by re-invoking the model with the same prompt, silently burning tokens. Explicit FSM design converts this to an exception.
Leaky tool permissions: An agent in the EXTRACTING state should not have access to the tools it needs in the ROUTING state. State-scoped tool injection — passing only the tools valid for the current state to the model — reduces both token cost and security surface. Most frameworks give the model the full tool list at every step.
Prompt drift across states: Using the same system prompt for all states produces mediocre performance across all phases. Each state has a different task. The EXTRACTING prompt should focus the model on structured extraction. The VALIDATING prompt should focus it on consistency checking. A single monolithic system prompt doesn't serve any state well.
Decision Criteria
When is a framework still the right choice? When the FSM is genuinely simple (2-3 states, linear progression) and the framework's ergonomics buy you meaningful development velocity without architectural lock-in. LangGraph is the one framework I currently recommend without reservation, because it makes the state machine explicit — you define the graph, the nodes, and the edges — rather than hiding it behind agent abstractions. If you find yourself using LangGraph's StateGraph, you're already thinking in FSMs. That's the right starting point.
Raw SDK plus approximately 200 lines of orchestration code is still my default recommendation for production systems where debuggability and operational cost matter more than development speed. Which, in my experience, is almost always.
Closing
The ReAct paper gave us the loop. The Berkeley leaderboard tells us how often the loop fails at each step. The FSM mental model tells us what to do about it. None of this is new — finite state machines have been the correct model for reactive systems for 60 years. The novelty is that the model's token generation provides the transition logic, which is probabilistic rather than deterministic. That probabilistic transition logic is why hard ceilings, explicit state tables, and escalation paths aren't optional — they're the load-bearing walls of any agent architecture that needs to survive contact with production.
Production Readiness Checklist
Before deploying any agent system built on explicit FSM design:
• [ ] Every state transition is logged with timestamp, input that triggered it, and model output that drove it
• [ ] max_turns is set and tested — verify the system reaches ESCALATING rather than running indefinitely
• [ ] Tool list is scoped per state — no state receives more tools than it needs
• [ ] TRANSITIONS table has been code-reviewed by someone other than the original author
• [ ] The ERROR state has an observable signal (alert, log, metric) that surfaces to on-call
• [ ] Retry logic has an explicit counter — no state can loop more than N times
What I Would Build Differently
The gap I most often discover in FSM-based agent systems is the absence of a transition audit log that's queryable by business users, not just engineers. When an agent escalates a task to a human reviewer, the reviewer needs to understand which state the agent was in, what it tried, and why it escalated. A transition audit log that surfaces in plain English ("The agent attempted extraction twice, reached low confidence on the termination clause, and escalated at turn 6") is dramatically more useful than a raw state history in a log file. Building it requires designing the state logging for human readability from the start, not retrofitting it after the first escalation confusion.
Figure 3. State machine diagram with seven nodes (INIT, EXTRACTING, VALIDATING, ROUTING, DONE, ESCALATING, ERROR) connected by labeled directed edges. Retry loop shown between VALIDATING and EXTRACTING with …
REFERENCES
1. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv / ICLR (2023).
https://arxiv.org/abs/2210.03629
2. Berkeley Function-Calling Leaderboard. Berkeley GORILLA Project (2024).
https://gorilla.cs.berkeley.edu/leaderboard.html
3. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023).
https://arxiv.org/abs/2302.04761
4. LangGraph Documentation: Stateful, Multi-Actor Applications. LangChain / LangGraph Docs (2024).
https://langchain-ai.github.io/langgraph/
5. An Introduction to Agent State Machines (Applied LLMs Guide). applied-llms.org (2024).
6. Anthropic Tool Use Documentation. Anthropic Developer Docs (2024).
https://docs.anthropic.com/en/docs/build-with-claude/tool-use
7. Constitutional AI: Harmlessness from AI Feedback. arXiv / Anthropic (2022).
https://arxiv.org/abs/2212.08073
8. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. arXiv / EMNLP (2018).
https://arxiv.org/abs/1809.09600
Comments (0)
Join the conversation!