Why GPT-5.6 Changed My Model Switching Strategy

My model router started making expensive mistakes the day GPT-5.6 hit production. Not because OpenAI changed their API—the endpoints worked fine. The problem was deeper: my carefully tuned switching logic was built on performance assumptions that GPT-5.6 made obsolete overnight.

The Router That Worked (Until It Didn't)

For three months, my model switching logic had been humming along predictably. I'd built what felt like a sensible system:

• Simple classification tasks → local Llama model (fast, cheap)
• Complex reasoning → GPT-4 (expensive but reliable)
• Speed-critical responses → Claude Haiku (good enough, very fast)

The logic was straightforward: match task complexity to model capability, with cost and speed as tiebreakers. I had spreadsheets showing consistent performance across my workload. Then GPT-5.6 launched and my neat categories started overlapping in ways that made my router look naive.

The first sign of trouble was my cost reports. Tasks I'd been routing to GPT-4 because "only it could handle the complexity" were suddenly being solved faster and cheaper by GPT-5.6. Meanwhile, my "fast" Claude Haiku routes were getting outpaced by GPT-5.6 on certain document types.

When Performance Assumptions Crumble

GPT-5.6's speed caught me completely off-guard. I'd categorized it as "the new expensive option" and routed accordingly. But it wasn't just faster than GPT-4—it was faster than Claude Haiku for structured document analysis, a task I'd never considered "complex reasoning."

The cascade effect was immediate. One changed performance variable broke multiple routing decisions. Tasks I'd been confidently sending to local models were failing at higher rates, while my expensive GPT-4 routes were solving problems that GPT-5.6 could handle for less money.

I realized I'd been optimizing for my mental model of how these systems worked, not for what they actually delivered on my specific tasks.

My A/B Testing Setup

Rather than guess at the new optimal routes, I set up proper A/B tests across my actual workload. I split traffic three ways for a month:

• Document analysis: Legal contract review, technical documentation parsing
• Code review: Pull request analysis, security scanning suggestions
• Content generation: Technical writing, API documentation

For each task type, I measured what actually mattered to me:

• Accuracy (successful task completion, not just API response)
• End-to-end latency
• Cost per successful completion (including retry costs)

I ran this across enough volume to feel confident—roughly 2,000 tasks per category, tracked over four weeks.

Surprising Results That Changed My Mind

The results broke several of my assumptions about model selection.

Document analysis showed the most dramatic shift. My local Llama model handled about 70% of cases perfectly well, but the remaining 30% created expensive downstream problems. GPT-5.6 caught edge cases that would send me into costly revision loops with the local model. The higher per-call cost was offset by fewer total calls needed.

Code review created an unexpected tie between Claude and GPT-5.6. Claude's thoroughness found more potential issues, but GPT-5.6's speed meant I could run multiple focused passes in the time Claude took for one comprehensive review. The total value delivered was surprisingly similar.

Content generation revealed the biggest flaw in my thinking. Claude Haiku was indeed fastest for first drafts, but GPT-5.6's output needed fewer revision cycles. When I measured cost per final deliverable instead of cost per API call, the economics flipped entirely.

The Cost Optimization Paradox

I kept finding cases where paying more per API call resulted in lower total project cost. This wasn't just about retries—it was about the hidden downstream costs I hadn't been tracking.

My "good enough" local model routes were generating technically correct responses that needed human cleanup around 40% of the time. That cleanup cost wasn't in my model comparison spreadsheets, but it was very real when I calculated total time-to-completion.

The switching logic I'd been proud of was optimizing for the wrong endpoint. I was minimizing API costs while accidentally maximizing total project costs.

Rebuilding My Router Logic

Instead of rule-based routing, I moved to dynamic performance thresholds based on what I'd learned from the A/B tests. The new logic considers:

• Task completion rate (not just API success rate)
• Total cost including revision cycles
• Confidence scores for routing decisions

I also started measuring "task completion" differently. A successful API call that produces output requiring significant human revision isn't actually successful for my purposes.

The router now tracks these metrics continuously and adjusts routing thresholds based on real performance data, not my assumptions about model capabilities.

What This Revealed About My Assumptions

I thought I understood the tradeoffs between speed, cost, and quality well enough to automate the decisions. But I'd been optimizing for theoretical model capabilities instead of real-world performance on my specific task distribution.

My "obvious" cost-saving measures—route cheap tasks to cheap models—weren't actually saving money when I factored in the full workflow. The local model that cost $0.02 per call but needed human cleanup 40% of the time was more expensive than GPT-5.6 at $0.15 per call with a high completion rate.

I also realized how much of my optimization was built on static benchmarks rather than dynamic performance measurement. The model that tested best on my evaluation set six months ago wasn't necessarily the model that performed best on yesterday's actual workload.

I'm still not sure I'm measuring the right things, or if the metrics that matter today will matter in six months when the next model launches. But I'm more confident that whatever assumptions I'm making now about optimal model selection will probably need updating sooner than I expect.