What I Found Running Smart Model Routing in Production for 3 Months

The $847 AWS bill that landed in my inbox three months ago made me question everything I thought I knew about model selection. I'd been routing every request through GPT-4 because the logic seemed obvious: better model equals better results. The bill suggested otherwise.

I wasn't just burning money on overkill—I was getting inconsistent results that didn't seem to correlate with the premium I was paying. Some of my simplest tasks were eating through tokens while producing output that felt no better than what I might get from cheaper alternatives. That's when I decided to build custom routing logic and actually measure what was happening.

Why I Built My Own Decision Tree

My application handles a mixed bag of work: content generation for marketing copy, data extraction from messy documents, code review summaries, and user query responses ranging from simple lookups to complex analysis. The pain point wasn't just cost—it was the nagging sense that I was using a sledgehammer for thumbtacks while occasionally needing that sledgehammer for actual sledgehammer work.

My hypothesis was straightforward: different tasks might perform optimally on different models regardless of their position in the supposed quality hierarchy. What I needed was data to prove or disprove this, which meant building routing logic that could track performance across multiple models for the same types of tasks.

The System I Built

The routing logic starts with task classification. I built a scoring system that evaluates incoming requests across three dimensions: complexity (keyword density, sentence structure, domain-specific terminology), required output format (structured data, creative text, code, analysis), and context requirements (how much background information the model needs to process).

My model pool includes Claude 3.5 Sonnet, GPT-4, GPT-3.5-turbo, and local Llama variants I've fine-tuned for specific domains. The decision tree routes requests based on cost thresholds, latency requirements, and accuracy minimums I've established for different task categories. When the primary model fails or hits rate limits, requests cascade through a fallback chain.

The interesting part was building evaluation criteria that could work across different model outputs. I settled on measuring task completion rate, output format compliance, and—where possible—downstream processing success (like whether generated code actually runs or extracted data passes validation).

Three Months of Surprising Results

The data revealed patterns I didn't expect. For structured output tasks—generating JSON from natural language descriptions, extracting specific fields from documents—GPT-3.5-turbo consistently outperformed GPT-4. Not just on cost efficiency, but on actual reliability. GPT-4 would occasionally inject creative interpretations or add fields I didn't request. GPT-3.5-turbo stuck to the script.

This happened across a significant portion of my structured tasks. The pattern was so consistent I started wondering if GPT-4's additional capabilities were actually working against it for deterministic outputs.

My fine-tuned local Llama model dominated data extraction tasks in my specific domain (financial documents). It beat both Claude and GPT variants on accuracy while running locally with zero API costs. The catch: it was useless for anything outside its training domain, but for the narrow slice it understood, it was untouchable.

The cost-per-quality math got interesting when I factored in latency. Premium models often delivered marginally better results but took 2-3x longer to respond. For user-facing queries where speed mattered, the "inferior" model frequently provided better user experience despite lower technical quality.

Counterintuitive Patterns

I found the creative versus analytical split everyone talks about doesn't hold up as cleanly as expected. Claude excelled at creative tasks but also showed surprising strength in logical reasoning chains. GPT-4 handled complex analysis well but sometimes overthought simple creative prompts, producing verbose outputs that needed editing.

Context length turned out to be a trap more often than a feature. Longer context windows didn't automatically improve performance—sometimes they hurt it. Models would get distracted by irrelevant information in large contexts, producing less focused outputs than when working with carefully curated shorter prompts.

The consistency factor emerged as crucial. Some models had higher peak performance but worse reliability. I'd rather have a model that produces solid work most of the time than one that produces exceptional work sometimes and fails spectacularly the rest.

My Current Decision Tree

After three months of iteration, my routing logic has settled into clear patterns. Simple classifications, data reformatting, and basic Q&A go straight to local models or GPT-3.5-turbo. Most content generation and complex but structured tasks hit the mid-tier models. I only route to frontier models for novel problem-solving, nuanced reasoning, or genuine edge cases where I need the extra capabilities.

The gray zones are what keep me experimenting. Tasks involving subtle context interpretation or domain boundary crossing still challenge my classification system. I'm constantly adjusting thresholds based on new data.

Unexpected Complications

Model personality quirks created cascade effects I hadn't anticipated. Claude's tendency toward verbose explanations worked great for some content types but broke downstream processing that expected concise outputs. GPT's more direct style was efficient but sometimes too terse for creative applications.

Prompt engineering became a nightmare when optimizing for multiple models. A prompt tuned for GPT-4's reasoning style would confuse GPT-3.5-turbo, while prompts that worked well for cheaper models didn't leverage the advanced capabilities I was paying for in premium models.

Caching introduced its own complexity. Smart routing sometimes hurt response times because switching between models meant more cold starts. The most efficient model for a task wasn't always the most efficient choice when factoring in infrastructure overhead.

Open Questions I'm Still Wrestling With

Measuring "better" across different task types remains challenging. Technical metrics like accuracy and format compliance are straightforward, but how do you quantify creativity or persuasiveness? I've started A/B testing outputs with end users, but their preferences don't always align with my technical measurements.

The rapid pace of model improvement makes me question how often I should revisit routing logic. My data is already three months old in a field where that might as well be three years. New model releases could invalidate my entire decision tree overnight.

What strikes me most is how little we actually understand about what makes one model output "better" than another for specific use cases. The data suggests our assumptions about model hierarchies might be more about marketing positioning than mathematical reality.