Frontend Beauty
ESC

Type to search the knowledge base.

Model Routing and Gateways

A gateway sits in front of many providers: auth, routing, fallbacks, and logging. Route simple tasks to small models and keep one SDK in the app.

intermediate6 min read
  • gateway
  • routing
  • llmops

Hard-coding gpt-something in fourteen route handlers is how a provider outage becomes a site outage. An LLM gateway sits in front of many backends: one auth story, one log, one place to retry, and a router that sends cheap tasks to small models. The app speaks one SDK. The gateway owns vendor drift.

This is the same instinct as a BFF. The browser talks to you. You talk to whoever is healthy.

Key takeaways

  • The application asks for a task (extract, chat, judge), not a vendor snapshot id, except where you pin for evals.
  • A gateway handles credentials, routing, fallbacks, rate limits, and traces. It is not “a smaller model” by itself.
  • Schema and tool compatibility are the real lock-in. A fallback that cannot call your tools is not a fallback.
  • Pin versions in CI and in traces. Aliases move.
  • Do not put the provider key in the browser and call that multi-cloud.

Why an LLM gateway exists

Without one, every Next.js route handler grows its own timeout, its own header, its own “if OpenAI 500 then…” You will get them wrong in different ways.

With one, the app does:

await gateway.complete({
  task: 'extract_invoice',
  messages,
  schema: InvoiceSchema,
  abortSignal,
});

The gateway does: pick a deployment, attach the server-side key, record usage, retry idempotent generations, fall back, export a span. LiteLLM-style proxies are this idea as a process you can run: OpenAI-compatible URL in, many providers out. You can also write a thin internal wrapper. The architecture matters more than the logo.

type Task = 'chat' | 'extract_invoice' | 'route_intent' | 'judge';

type ChatMessage = { role: 'system' | 'user' | 'assistant' | 'tool'; content: string };
type ToolSchema = { name: string; parameters: Record<string, unknown> };
type CompletionResponse = { text: string; model: string };

type Deployment = {
  id: string;
  task: Task;
  provider: 'openai' | 'anthropic' | 'google' | 'local';
  model: string; // pinned snapshot
  order: number; // lower is tried first
  supportsTools: boolean;
  supportsJsonSchema: boolean;
};

type GatewayRequest = {
  task: Task;
  messages: ChatMessage[];
  tools?: ToolSchema[];
  abortSignal?: AbortSignal;
};

class ModelGateway {
  constructor(
    private deployments: Deployment[],
    private completeWith: (d: Deployment, req: GatewayRequest) => Promise<CompletionResponse>,
  ) {}

  async complete(req: GatewayRequest): Promise<CompletionResponse> {
    const candidates = this.deployments.filter((d) => d.task === req.task);
    const ordered = this.rank(candidates, req);
    let lastErr: unknown;
    for (const d of ordered) {
      if (req.tools?.length && !d.supportsTools) continue;
      try {
        return await this.completeWith(d, req);
      } catch (err) {
        lastErr = err;
        if (!isRetryable(err)) throw err;
      }
    }
    throw lastErr ?? new Error('no deployment');
  }

  private rank(ds: Deployment[], _req: GatewayRequest): Deployment[] {
    // Order is config, not a call-site model id.
    return [...ds].sort((a, b) => a.order - b.order);
  }
}

class HttpError extends Error {
  constructor(public status: number, message: string) {
    super(message);
  }
}

function isRetryable(err: unknown) {
  return err instanceof HttpError && (err.status === 429 || err.status >= 500);
}

Keep provider keys on the server. The browser streams from your origin. See client vs server LLM calls.

Task-based routing

Routing is a product decision table, not a magic classifier you cannot debug.

Task Default Fallback Why
Intent / classification Small, fast model Another small model Volume, latency
Structured extract Model with JSON schema Same family, pinned Validity + eval
RAG answer Mid-size, tools off Larger only if eval requires Cost vs groundedness
Agent with tools Model that actually calls tools Provider with the same schema Compatibility
Judge Separate pinned model None (fail the eval run) Do not silently change scores

A classifier that reads the user message and picks “easy vs hard” is allowed if you log the choice and eval both paths. A hidden “auto” router that you cannot replay is how quality regressions vanish.

Config belongs outside the UI code. Changing “extract uses model B” should not require a frontend deploy. Changing the task a screen calls might.

Compatibility traps

Fallbacks fail in ways load balancers do not:

  • Tool schema dialects. OpenAI-style function tools vs provider-specific. If the backup cannot see refund_order, it will narrate instead of call.
  • Structured output. One backend constrains decoding to JSON Schema. The backup returns prose. Your Zod parse fails closed (good) and the user sees an error (you needed a second compatible backend, or no fallback).
  • Context windows. Routing a 100k-token pack to a 32k model is not a fallback. It is a 400.
  • Multimodal. A text-only backup cannot “see” the screenshot. Refuse, do not pretend.
  • System prompt + cache. Different providers cache differently. A prefix tuned for one may miss on another. Measure per deployment.

If the task needs tools, the candidate list must be supportsTools: true. Skipping that check is how you “fail over” into a chatbot that apologizes.

Fallbacks, cooldowns, and what the app still owns

Gateway retries belong on generation, not on tools that charge cards. Cooldown a deployment after a streak of 429/5xx so you do not hammer a sick region. Cap fallback depth. An infinite walk across models is a latency bug with extra invoices. Details live with retries, timeouts, and fallbacks.

The app still owns:

  • Per-user quotas (the gateway will happily spend your key).
  • Abort from the UI.
  • Prompt versions and eval pins. Production can route; CI should call a pinned deployment so scores do not drift.
  • Traces: deployment.id, model.id, task.

Cost optimization without a gateway is copy-paste. With a gateway, “send classify traffic to the small model” is a config change you can roll back.

One more product rule: the UI should not branch on vendor names. If extract fails schema validation, show an extract error, not “Anthropic is down.” The gateway can attach deployment.id to the trace and to an internal header. The composer stays vendor-blind. That is how you avoid painting a provider logo onto a timeout.

Interview answer (out loud)

“An LLM gateway is a server-side proxy: one SDK in the app, credentials, routing, retries, and logging behind it. I route by task, not by sprinkling model ids in fourteen handlers, and I only fall back to deployments that support the same tools and schema. CI pins snapshots. The browser never sees a provider key. A gateway does not replace evals; it makes changing the model a controlled operation.”

Further reading (primary sources)

Related guides