Running My Own Smart Model Router After Claude's Failed My Production Workload
Three weeks ago, I started getting complaints from users about "weird AI responses" in the dashboard I built for a client. The complaints were vague but consistent — outputs that didn't match the input, responses that seemed to ignore half the context, occasional gibberish mixed with perfectly coherent text.
I'd recently switched to Claude's smart routing feature, which automatically selects the best model for each request. It looked perfect in my testing: clean API, automatic model selection for my mixed text/image/code pipeline. The integration took an afternoon, and the demo scenarios ran flawlessly.
Took me a full week to realize the router was silently failing on a significant portion of our multi-modal requests.
The Silent Failures
The production pipeline I built processes a mix of text analysis, image interpretation, and code review requests. Users upload everything from customer feedback screenshots to technical documentation with embedded diagrams. During development, I tested with clean examples — well-formatted images, standard text lengths, predictable combinations.
Claude's smart routing handled these perfectly. The marketing materials promised it would "intelligently distribute requests to optimize for quality and speed." The API documentation showed simple examples that worked exactly as advertised.
But my real users don't send clean examples. They upload blurry phone photos of whiteboards, paste in text with mixed languages and unusual formatting, send lengthy technical documents followed immediately by single-sentence questions. The router that worked flawlessly in testing was choking on the messy complexity of actual usage.
The failure mode was particularly insidious — no error messages, no obvious breaking points. Just degraded quality that users noticed but couldn't articulate clearly.
The Debugging Nightmare
Claude's routing logic is completely opaque. No logs showing which model handled which request. No documentation of failure modes or edge cases. No visibility into why certain inputs get routed where.
I spent three weeks trying every combination I could think of:
- Different prompt formats and payload structures
- Various timeout and retry settings
- Restructuring how I batched multi-modal requests
- Breaking complex requests into smaller chunks
Support tickets got me generic responses about "optimizing prompts for better routing performance." The suggestions were clearly copy-pasted from documentation I'd already read.
The most frustrating part: the failures weren't random. They clustered around specific patterns that my users actually send — technical documentation with mixed content types, images with overlaid text in non-English languages, requests that combined visual analysis with code generation.
These weren't edge cases I was manufacturing to stress-test the system. This was the core of what my application needed to handle.
What Smart Routing Actually Optimizes For
Looking at vendor demos and documentation, I started noticing a pattern. Every example shows clean, predictable inputs that route perfectly:
- Single-language text with standard formatting
- High-quality images with clear subjects
- Well-structured code in popular languages
- Requests that fit neatly into obvious categories
My real data looked nothing like this. Users upload photos taken in poor lighting, paste text from PDFs with formatting artifacts, send messages that mix technical jargon with casual language. The router that handled tutorial examples gracefully was breaking on the complexity of actual usage.
I realized the smart routing was optimized for broad adoption across many use cases, not deep reliability for any specific one. It's designed to work well enough for most customers most of the time, not to handle the specific edge cases that make or break individual applications.
The abstraction was solving for Anthropic's scaling problem — how to serve millions of diverse requests reasonably well — not my reliability problem.
Building Deterministic Logic
Four days after I gave up on debugging Claude's router, I had my own routing system working better than the smart version ever did.
The logic is embarrassingly simple:
- Check for image content and analyze complexity
- Measure text length and identify language patterns
- Consider user context and response time requirements
- Route based on explicit rules I can modify and debug
I added explicit fallback chains for when the primary model is unavailable, detailed logging for every routing decision, and error handling that actually tells me what went wrong.
Now when something breaks, I can trace exactly why a request went to a specific model and adjust the logic accordingly. The routing rules encode my actual domain knowledge about user patterns and quality requirements.
The code is maybe 200 lines. The debugging cycle went from weeks to minutes.
Why Abstractions Break at the Edges
Smart routing promises to eliminate decisions I actually need to make about my specific use case. But those decisions contain domain knowledge that no general-purpose system can replicate.
My routing logic knows that users in certain workflows prioritize speed over perfect accuracy. It understands that technical documentation requests need different handling than casual chat messages. It accounts for the specific failure patterns I've observed in my user base.
The vendor abstraction optimizes for the average case across all their customers. My hand-written logic optimizes for the specific edge cases that matter most to mine.
This isn't a critique of Anthropic's engineering — they're solving a different problem than I am. They need horizontal scaling across diverse use cases. I need vertical reliability for one specific application.
The Real Cost of Black Boxes
I lost three weeks of development time on what I thought was a "solved" problem. Users experienced degraded quality during that entire period, with no clear way for me to communicate why or when it would improve.
The cognitive load of not understanding failure modes turned out to be higher than just writing the routing code myself. Every user complaint became a mystery to solve instead of a bug to fix.
I'm now maintaining routing logic that I could have written from the beginning, but I understand exactly how it works and can modify it when requirements change.
The abstraction promised to save me time but ended up costing more than building the functionality directly.
When Abstractions Make Sense
I'm not arguing against vendor solutions entirely. Claude's actual language models are abstractions I'm happy to use — the complexity of training and serving large language models is genuinely beyond what I want to manage.
But routing logic sits at a different layer. It encodes business rules and domain knowledge that are specific to my application. The decision of which model to use for which request depends on context that only I have access to.
The more domain-specific the problem, the less likely a general-purpose abstraction will handle your edge cases reliably.
I'm still wondering when I started trusting vendor abstractions more than my own ability to solve problems I actually understand.