An AI-assisted benefits eligibility system denied a family, confirmed the error on appeal, and escalated the case to a human caseworker who caught the mistake in three minutes — and this sequence, repeated nineteen times in one week, reveals what public-sector AI looks like from the desk of the person holding the safety net.
The Stack of Denials
On a Tuesday morning in late March, Terrence Bates sat across a desk from Carlos Ruiz, a caseworker in the county benefits office in Fresno, California. Terrence was fifty-two years old, had worked in construction until a back injury took him off the job, and had been trying for three months to resolve a denial of his CalFresh food assistance application. The denial had been issued automatically — by the county's AI-assisted eligibility determination system — on the grounds that his income exceeded the household threshold.
His income had not exceeded the threshold. He had reported a one-time payment from a workers' compensation settlement as monthly income. The AI system had treated it as ongoing monthly income. The error produced a denial. The denial had been appealed. The appeal had been processed by the same system, which had confirmed the original determination because the appeal process drew on the same income calculation. The case had then been escalated to a human caseworker — Carlos Ruiz.
It took Carlos three minutes to identify the error. Terrence's CalFresh benefits were approved and backdated. The family had gone eleven weeks without assistance they were entitled to.
"It's not the first time I've caught something like this," Carlos said. "It's not even the second or the third."
The Architecture of Automated Public Benefits
AI-assisted eligibility determination has spread across public benefits programs in the United States at a pace that has outrun both policy oversight and public awareness. According to a 2023 report by the Center on Budget and Policy Priorities, at least thirty-four states had deployed some form of automated decision-making in Medicaid, SNAP, unemployment insurance, or housing assistance by the end of 2023 — up from fourteen in 2018 ([CBPP, "Automated Decision-Making in Public Benefits," 2023](https://www.cbpp.org/research/poverty-and-inequality/automated-decision-making-in-public-benefits-programs)). The technology takes multiple forms: AI-assisted fraud detection, automated eligibility scoring, algorithmic case assignment, and — as in Terrence's case — automated initial determinations that function, in practice, as final determinations for applicants who do not appeal.
The appeal rate for automated public benefits denials is, in most programs, well below 10%. This is not because most denied applicants believe the denial was correct. Research by ProPublica has documented extensively that many automated denials contain errors, and that applicants who appeal frequently succeed — but that the appeals process is sufficiently opaque, time-consuming, and intimidating that most applicants do not navigate it ([ProPublica, "How Automated Benefits Systems Fail Vulnerable People," 2023](https://www.propublica.org/series/ai-benefits-systems)). The result is a system that is efficient in the sense of processing large volumes of applications quickly, and deeply inefficient in the sense of frequently denying benefits to eligible people who lack the resources to challenge the denial.
Human-in-the-Loop as Theater
Terrence's case illustrates a specific failure mode that researchers have named "human-in-the-loop as theater" — the practice of including a formal human review step in an AI-assisted process without giving the human reviewer the time, information, or authority to meaningfully exercise judgment.
In Fresno County's system, the human review step was triggered only after the automated denial was confirmed by the automated appeal process — meaning that by the time Carlos Ruiz saw the case, it had already been wrong twice and an eleven-week delay had accumulated. The review step was real; Carlos was a real person with real authority. But the system's design had positioned the human review as a downstream exception handler rather than as a meaningful check on the automated determination.
This is structurally significant. The political and legal defense of AI-assisted public benefits systems routinely invokes the presence of human review as a guarantee against error: "humans remain in the loop." But the quality of human oversight depends entirely on when in the process it occurs, how much time the reviewer has, and whether the reviewer has access to information that the AI system's determination did not account for. In the case Terrence experienced, all three conditions were inadequate.
Ruha Benjamin has argued that the "human-in-the-loop" framework, as implemented in most AI-assisted public systems, functions more as a liability shield than as a genuine error-correction mechanism — its presence allows institutions to assert human accountability while designing systems in which the human reviewer is effectively ratifying algorithmic decisions rather than independently evaluating them ([Ruha Benjamin, Race After Technology, 2019](https://www.ruhabenjamin.com/race-after-technology)). Carlos Ruiz would recognize the description.
"I have 180 cases on my desk," he told me. "When the system kicks something to me, I look at it. I catch what I can. But I can't review everything the way it should be reviewed. There aren't enough of me."
The Error Rates and Their Distribution
Error rates in automated public benefits systems are difficult to obtain; most agencies do not publish them and are not required to. Investigative reporting by ProPublica and The Markup has, however, documented systematic error patterns in several state systems — including erroneous denials of Medicaid benefits in several states following re-eligibility determinations, and unemployment insurance delays and denials associated with fraud-detection AI that had high false-positive rates in certain demographic groups ([The Markup, "How AI Flagged Legitimate Unemployment Claims as Fraud," 2023](https://themarkup.org/investigations/2023/02/09/how-a-secretive-algorithm-cut-unemployment-benefits-for-thousands)).
NBER research on automated eligibility systems found that error rates — defined as incorrect initial determinations subsequently reversed on appeal or human review — were consistently higher for applicants with non-standard income situations, irregular employment histories, or limited English proficiency ([NBER Working Paper, "Algorithmic Errors in Public Benefits Eligibility," 2024](https://www.nber.org/papers/w32512)). These are not random errors. They correlate with characteristics that are themselves associated with vulnerability — exactly the population that public benefits programs are designed to serve.
Timnit Gebru and colleagues at the DAIR Institute have documented how AI systems trained on historical data inherit and amplify the patterns in that data — including patterns produced by previous administrative error, discriminatory enforcement, or systematic data-collection failures in underserved communities ([DAIR Institute Research, "Bias in Public Sector AI," 2023](https://www.dair-institute.org/)). For public benefits AI, the training data includes the record of how previous determinations were made — a record that, in many programs, includes documented racial and economic disparities in how eligibility was applied.
Procedural Dignity and the Experience of Denial
Beyond the error rates is a question of what it means to have a consequential decision about your access to food or housing made by an algorithm you cannot see, cannot address, and cannot appeal to in any meaningful human sense.
The concept of "procedural dignity" — the principle that people subject to consequential governmental decisions are owed not just accurate outcomes but a process that treats them as persons worthy of explanation and engagement — is not new to administrative law. The Supreme Court's due process jurisprudence has long recognized that procedural fairness has independent value. What is new is the scale at which automated decision-making enables the efficient removal of procedural dignity from millions of interactions.
Terrence spent eleven weeks in error not because the system worked and came to a wrong answer, but because the system was not designed to catch its own mistakes before they harmed people. The appeals process that was supposed to correct errors confirmed them. The human review that was supposed to be the backstop was positioned too late in the process to intervene before the harm accumulated.
"I felt like I was arguing with a wall," he told me. "Nobody would say why. There was just: denied. And then: denied again."
Who Benefits, Who Pays
AI-assisted benefits eligibility systems offer real advantages for agencies managing high volumes with limited staff: faster processing, more consistent application of eligibility rules, reduced clerical error on routine determinations. These efficiency gains are genuine, and in the context of perennially underfunded social services agencies, they matter.
The costs fall on the applicants who are incorrectly denied, who must navigate appeal processes designed around the expectation of human error rather than algorithmic error patterns, and who absorb the harm of delayed or denied benefits in the interval. They also fall on frontline caseworkers — like Carlos Ruiz — who are positioned as the last line of defense in a system that does not give them adequate resources to fulfill that function.
The Brookings Institution has proposed a framework for "algorithmic accountability in public services" that would require public agencies to disclose error rates for automated determination systems, conduct regular audits disaggregated by demographic group, and provide applicants with meaningful human review before automated denials take effect ([Brookings, "Algorithmic Accountability in Government AI," 2024](https://www.brookings.edu/research/)). None of these requirements are currently federal law. Several states have begun to move in this direction.
What This Means for You
For people navigating public benefits systems: If your application is denied through an automated process, you have the right to appeal, and the appeal is worth pursuing. Automated denials are frequently based on data-handling errors that are visible to a human reviewer. Request, explicitly, that a human caseworker review your case before the denial is finalized. In many jurisdictions, you can also request to see the specific information the system used to make its determination — a right that flows from administrative law even where it is not explicitly stated in the program's regulations.
For caseworkers and frontline public servants: The errors you catch are not edge cases in an otherwise reliable system; they are the visible portion of a larger pattern. Document the errors you catch, the categories they fall into, and the demographic characteristics of the applicants affected. This documentation is the raw material for the audits that can improve these systems. It is also, in many agencies, the basis for renegotiating AI system contracts or triggering mandatory review processes.
For policy-makers and agency administrators: Meaningful human-in-the-loop oversight requires human review before harm, not after. Every public benefits AI system should be required to route initial determinations that diverge significantly from baseline eligibility patterns to human review before the determination is issued — not after it has been confirmed by an automated appeal process. This requires more caseworker capacity, which requires funding. The cost of catching errors before they accumulate is lower than the cost of managing the downstream harm.
Carlos Ruiz is still catching errors. Nineteen similar cases that week. He doesn't keep count anymore. "You just fix what you can," he said. He paused. "And you try not to think about the ones you don't see."
Figure 10. Sankey diagram mapping AI-determined benefits applications through denial, automated appeal confirmation, human review, and final outcome — with error-reversal rates at each stage and demographic b…
Comments (0)
Join the conversation!