In short
Not every question needs the heaviest AI model. With model escalation, a question first goes to the smallest model that can probably handle it. A second model checks the answer. If that answer is not yet certain and complete, the system scales up: it looks for more evidence, or has a larger model improve the answer.
How it works
- Assess. Before anything is answered, the system determines what kind of question it is: short or complex, closed or open, with or without current information. Anything that can be calculated or looked up is done by the code itself: deterministic first.
- Route. The question goes to the smallest model that fits. That is fast and cheap.
- Check. A checking model gives the answer a score. Because an AI model does not say the same thing every time (nondeterminism), when there are several answers we have another model check them. We look at the cross-section of the outcomes.
- Scale up or finish. If the score is too low, the system chooses the best next step: search more broadly in your own sources, look something up in a targeted way, or have a heavier model revise the answer. If only the user can supply what is missing, it asks one follow-up question.
Why this works
Most questions in a business can be answered well by a small model, provided it gets the right information. In our own measurements, a question that went wrong with an entire case file as context went right when the model was given only the few relevant facts, and it was many times faster as well. This measurement shows that searching more precisely is sometimes enough, and scaling up is not needed.
What to watch out for
- Define when an answer is "good enough". A closed question demands certainty, an opinion does not.
- Make every step visible: which model, which score, why it scaled up. Then it is not a black box.
- Better an answer with an honest note of doubt than a confident guess. That is how you keep hallucinations in check.
Model escalation makes local AI on your own hardware feasible: the heavy models only run when they are really needed.
Would you like advice on choosing AI models? Request a free advice conversation.
Updated on