Model Routing and Fallbacks

Model Routing and Fallbacks

Step 1 of 9

1,000 req$15.00avg1.8 s

Without a router every request goes to the large model: 1,000 of them cost $15.00 and wait 1.8 s each.

The Idea

Most traffic is easy: summaries, classifications, short lookups. A router scores each request: easy ones go to a small, cheap, fast model, hard ones to a large one. A fallback catches the misses: when the small model's confidence is low, or a call fails, the request moves up a tier instead of retrying the same one.

Real-World Example

A print shop sends business cards to the office printer and a thousand-page catalogue to the industrial press. A job the office printer chokes on goes to the press, not back into the same tray.

The Tradeoff

Every escalation pays for both models, and the router itself has to cost less than it saves. Track quality and escalation rate per tier, not just the bill. On Amazon Bedrock, intelligent prompt routing does this between two models of one family, with a fallback model as the anchor.

Your turn

Put the steps in the right order.

1. 1Escalate low-confidence answers to the larger model↑↓
2. 2Score how hard the incoming request is↑↓
3. 3Send it to the cheapest tier that can handle that score↑↓
4. 4Review cost, quality and escalation rate for each tier↑↓

Check order

Mini quiz

1 / 3

A router sends 700 of 1,000 requests to a small model. For the router to pay off, each routing decision must cost:

• ALess than the price gap it saves on the requests it diverts
• BNothing at all
• CMore than the large model, so it is accurate
• DExactly what the small model costs