My city's 311 system routes everything to a human dispatcher. I scraped 18 months of requests, trained a classifier, and built a chat layer. The routing accuracy is better than the baseline. The city wants in.
|
THE STACK • Mistral 7B • Unsloth • FastAPI • Postgres • Railway |
Eighteen Months of "The Streetlight Is Out"
Here's what I just tried: I filed a 311 request to report a pothole in front of my apartment. Six days later, it had been routed through three departments and was sitting in a maintenance queue labeled "general infrastructure." The pothole was still there. I did what any reasonable person would do: I filed a public records request for the last 18 months of 311 data.
What came back was a 340MB CSV with 127,000 requests. And the problem was obvious — 34% of requests were labeled "general" because the dispatcher had categorized them broadly rather than drilling into the correct subcategory. "General" is not a routing category. It's a shrug.
I spent a three-day weekend building a better classifier. Then I emailed the city's Chief Innovation Officer. She wrote back in four hours.
The Stack
• Mistral 7B — base model for fine-tuning; strong classification performance, runs on a single A10G
• Unsloth — fine-tuning with LoRA; 3x faster training than vanilla HuggingFace, much lower VRAM
• FastAPI — the API layer; accepts a request description, returns a category prediction with confidence score and routing department
• Postgres — stores training data, model predictions, and feedback loop corrections
• Railway — deployment; one railway up command, API live with a public URL
The Data Work
I dropped all 127,000 records labeled "general" — uncertain labels train uncertain models. That left 84,000 records with 47 specific categories. I consolidated those into 12 routing categories mapping to actual city departments (Public Works, Parks, Housing, Transportation, Utilities, etc.) — many of the 47 were redundant. This taxonomy simplification took two hours with a spreadsheet and the city's public org chart.
Split: 70k training, 7k validation, 7k test. Fine-tuned Mistral 7B with Unsloth LoRA for 2 epochs on a Modal A10G. Training time: 3 hours. Cost: $4.20.
The Chat Layer
A plain classifier isn't citizen-facing. I added a FastAPI endpoint that wraps the classifier with a clarifying-question layer: if confidence is below 0.80, the API returns a question rather than a routing decision. "Is the streetlight completely out, or is it flickering?" routes differently than a dead light.
The API returns JSON with: predicted_category, confidence, routing_department, clarifying_question (null if confidence is high), and estimated_response_time (a lookup from historical response times in Postgres by category).
Where It Almost Went Wrong
The model badly underperformed on multilingual inputs. My city has a significant Spanish-speaking population, and about 8-10% of requests are in Spanish — but only 3% of the training data was Spanish. The model defaulted to "Transportation" as a catch-all for Spanish inputs.
Fix: back-translation augmentation — ran the Spanish training examples through Claude's translation API to create English versions, then translated 500 English examples per category into Spanish to boost Spanish representation. Spanish classification accuracy went from 61% to 84%. Not perfect, but a significant improvement for a weekend patch.
Benchmark vs. Baseline
Model accuracy on the test set: 91.3% across 12 routing categories. The city's estimated baseline routing accuracy on non-"general" requests: about 78% (based on the re-routing rate in historical data). On the formerly-"general" requests, the model classifies 88% with high confidence into a specific routing category.
The Chief Innovation Officer's team wants a 90-day pilot: every incoming request goes through the model, and the dispatcher sees the suggestion alongside the free-text request. Overrides feed back into training data.
Try This
1. File a public records request for your city's 311 data — most cities are required to provide it, and it's often already on open data portals. This is your free training dataset.
2. Consolidate categories to match actual routing destinations, not how the data is labeled. The routing taxonomy matters more than the classification taxonomy.
3. Fine-tune Mistral 7B with Unsloth on Modal — for 70-80k examples, two epochs is enough. Budget $5-10 for training experiments.
4. Add a confidence threshold and clarifying question path — below 0.80, ask rather than guess. Especially important for citizen-facing applications where misrouting is a trust failure.
5. Build the feedback loop from day one — store every prediction and every human correction in Postgres. That correction data is your next training set.
DIAGRAM_HINT: system diagram showing citizen 311 chat input → FastAPI classifier endpoint → Mistral 7B confidence check → high-confidence routing to city department or clarifying question loop → Postgres prediction log and human correction feedback → retraining pipeline
Figure 10. system diagram showing citizen 311 chat input → FastAPI classifier endpoint → Mistral 7B confidence check → high-confidence routing to city department or clarifying question loop → Postgres predict…
Comments (0)
Join the conversation!