LLM gateways compared: choose where model traffic is controlled

Name the decision the gateway has to make for you

An AI gateway sits between your application and the model providers it calls. Before you compare products, write down which request decisions you want to move out of application code. Common ones: which model or provider serves a request, what happens when that provider returns errors, how much each team may spend, which responses can come from a cache, and where the request log lives. A gateway that is strong on one of these can be thin on another, so the list decides the shortlist.

The cost explainer covers why model spend grows in agent systems. This guide covers the product decision. The documented workflows in the table come from the linked primary documentation, checked October 25, 2026. The fit questions and the pilot are our editorial recommendations. The five options illustrate different starting points; they are not a complete inventory or a ranking, and this guide reports no measured latency or reliability.

Compare who runs it and whose accounts it bills

Two questions separate these options more than any feature list. Who operates the gateway: your team, the vendor or a network you already use? And whose provider accounts pay for the tokens: yours, through keys you supply, or the gateway vendor’s, through credits you buy from it?

LiteLLM: the documentation describes two pieces. The Python SDK calls many providers through one completion() interface, and its Router adds retries, fallbacks and load balancing across deployments. The proxy is a self-hosted, OpenAI-compatible gateway, so any client that works with OpenAI’s API can point at it. The proxy page describes virtual keys for each user, team and project, each with its own models, budgets and rate limits, plus spend tracking and an admin UI. The repository license is MIT outside the enterprise/ directory, which has its own license.

OpenRouter: the quickstart shows one endpoint for many providers’ models, usable from the OpenAI SDK by changing the base URL. The provider routing guide says requests are load balanced across providers with price as the default priority, that you can sort by throughput or latency instead, that backup providers are allowed by default, and that you can restrict routing to zero-data-retention endpoints. The FAQ says OpenRouter passes through provider pricing with no inference markup, charges a fee when you buy credits, and does not log prompts or completions unless you opt in. Bringing your own provider keys is free up to a monthly allowance, with a 5 percent fee above it.

Portkey: the gateway repository is MIT licensed and runs locally with npx, in Docker or on Portkey’s cloud. Its README lists fallbacks on chosen error codes, up to five retries with exponential backoff, weighted load balancing across keys or providers, and pre-built guardrails that check inputs and outputs. The AI gateway documentation adds simple and semantic caching. The README marks some features as available only in the hosted and enterprise versions, and it announces a Gateway 2.0 pre-release that merges the enterprise gateway into open source.

Cloudflare AI Gateway: the documentation lists analytics, logging, caching, rate limiting, request retries and model fallbacks for calls to Workers AI and providers such as OpenAI, Anthropic and Google Gemini. The pricing page says the core features are free with a Cloudflare account. Accounts whose first gateway was created on or after September 24, 2026 use Workers Logs pricing and retention; older accounts use Legacy Logs with storage limits by plan. Log exports follow Logpush pricing, and credits bought through Unified Billing carry a 5 percent fee.

Kong AI Gateway: the documentation describes routing with failover and load balancing across providers, access control, and analytics for requests, tokens, errors and latency, exported through metrics, audit logs or OpenTelemetry. It covers MCP and agent-to-agent traffic as well as model calls. The quickstart creates a control plane in Kong’s Konnect service with a local data plane in Docker, and a separate path runs AI Gateway on self-hosted Kong Gateway with AI plugins.

Conor worked at Galileo, and Galileo sponsored the show’s first season, most of its second and the first fifteen episodes of its third.

Run a provider-outage pilot

Suggested pilot, not a reported benchmark: send the same staging traffic through each candidate, with two providers configured for one model family.

Pilot request	What to check
The primary provider returns rate-limit errors	Whether the gateway retries, falls back to the second provider, and records which provider answered
A team exceeds its budget mid-day	Whether the next request is blocked, warned or allowed, and who gets notified
The same prompt arrives twice	Whether the second answer comes from cache, and whether a cached answer is ever wrong for the user who receives it
A request contains personal data	What the gateway logs, where the log is stored and how long it is kept
A streaming response is cut off	How the client sees the error and whether the partial response is billed
The gateway itself is unreachable	Whether your application fails closed, fails open to a direct provider call, or queues the request

Check that the fallback model can do the job. A fallback to a cheaper model may return an answer your evaluation suite would fail. Decide per route whether a lower-quality answer beats an error.

Count the hops and the copies

Every gateway adds a network hop and a place where prompts and responses can be stored. Measure the added latency from the regions you serve, and confirm what the gateway logs by default. A hosted gateway that buys model access for you also becomes a party to your provider relationship, so check its data terms alongside the providers’. If you only need request logs, a gateway’s logging may overlap with an observability tool you already run; pick one place to keep the record. For routing between models by task difficulty, see model routing.

Where the interviews fit

Vercel CTO Malte Ubl described Vercel’s AI Gateway as a way to use every provider’s models without separate API keys once configured, framed mainly as a convenience during development. He said production access to the same models was billed at market rate. Redpanda CTO Tyler Akidau listed LLM gateways and MCP gateways among the infrastructure his company provides for agents. These are the guests’ views. Both companies sell gateway products not covered in the table above. Episode 39 was sponsored by Galileo while Conor worked there. Conor said at the start of episode 60 that no presenting sponsor had been secured as of recording.