Models get deprecated on a vendor’s timeline, but you’re the one who feels the impact. If a model gets pulled overnight for legal reasons, or refuses to generate an output mid-generation, it’s your product that suffers. Any solution hard-wired to one model has coupled its uptime to a supplier it doesn’t control, and can’t expect that supplier’s stability to hold.
That’s why at Edge Delta, we built AI Teammates to be model agnostic. They can connect to any model, treating each one, Claude, OpenAI, Gemini, Grok, and others, as a swappable, cost-aware, fail-safe component behind a single interface.
In this post, we’ll get into why that matters, and how we built it.
The fastest way to get stranded
Import one vendor SDK and call it. It’s the fastest way to ship and the fastest way to get stranded: the day that model is deprecated, refuses a legitimate request, or gets embargoed, your agent is down, and retrofitting an abstraction across a codebase that assumed one message shape, one tool-call format, and one error vocabulary is a painful migration under pressure, exactly when you have no time for it.
One registry, three axes
Every model the product can run is a single registry entry that captures everything the runtime needs to treat it uniformly:
// the model registry, one entry per model
{
provider // model provider, such as AWS Bedrock Mantle
owner // model owner
perTokenCost // token cost, tracked separately for input and output tokens
maxTokenWindow // maximum context window the model supports
possibleErrors // taxonomy of error classes with regexes, so provider errors surface to users in a friendly way
deprecation // whether the model is deprecated, and which model to continue with instead
fallback // for some models (such as Claude Fable 5), a pre-defined model to fall back to when the provider triggers it
}The rest of the codebase talks to a single Completions interface and never sees a vendor SDK. Three separate axes do the work: who made the model (owner), which provider we route through, and which API transport it speaks. That separation is what lets us add xAI’s Grok through an AWS Bedrock gateway that mimics OpenAI’s API shape, authenticated with SigV4 where an API key normally goes. One vendor’s model, reached through a second vendor’s cloud, speaking a third vendor’s protocol. The agent loop above it doesn’t know or care.
Routing for cost, failing over on refusal
On top sits cost-aware routing. Before the loop runs, a small fixed classifier decides whether a cheaper secondary model can fully handle this turn: route definitions and single lookups down, keep cross-signal root-cause analysis on the frontier model. The decision logic fails closed: if it errors or times out, we keep the primary model. When in doubt, keep the primary is written into the prompt, because a wrong downgrade on a live incident is expensive and hard to detect.
When a model refuses mid-generation, we detect the actual stop reason (which every provider buries in a different corner of its response metadata), fail over to the model’s declared fallback target if available, and pass the available fallback credit token back on the retry (if possible) so the fallback inference isn’t billed twice. The whole switch is a durable step on Restate and is surfaced to the user as a “switched model” pill, so nothing happens invisibly. And when we had to disable a model overnight for a US-Government action, it was a four-line change: flip its deprecation pointer to a replacement. That’s the payoff of the abstraction being there before you need it.
Context windows and structured output are per-model, too
The registry earns its keep in quieter ways. Long-context survival, the backbone of multi-hour investigations, is gated per model: the providers that support it get beta headers enabling automatic clearing of old tool results at 80K input tokens, while every model benefits from app-level summarization that compresses any tool output over ~40K characters with a cheap model before it ever hits the prompt. Structured output is per-model as well, and it’s not uniform: some providers do native strict-schema output, and at least one major one throws if you ask for it. So the registry records each model’s capability and we route structured requests accordingly. “Which model” is never just a name; it’s a bundle of capabilities the runtime has to reason about.
What model independence really takes
The abstraction sounds like a switch statement. It isn’t. Every provider differs in message shape, tool call format, error vocabulary, temperature semantics, and structured output support (some can’t do native structured output at all), and each one hides a refusal somewhere different. Cost routing is a non-deterministic decision, and it has to fail closed, otherwise it quietly downgrades an incident without telling anyone. Fallback has to be billing correct and durable, or it’s not really fallback.
Teams build the switch statement and think they’re done. Then a provider ships a breaking response change, or deprecates a model with 30 days notice, or an inference call refuses in a way their code doesn’t recognize, and they find out the abstraction they skipped is now a cross cutting migration, usually under a deadline. We paid that cost once, up front, and on purpose.
To see how it works in practice, try out Edge Delta’s AI Teammates for free and deploy it in minutes.
