A company has a dozen teams calling three model providers, each with its own API key, budget and logging. Finance can’t see total spend, security can’t see what data leaves the building, and an outage at one provider takes down every feature that uses it. Putting a gateway in front changes that: apps send every request to one endpoint, and the gateway holds the keys, picks the model (model routing), retries on another provider when one fails, blocks requests over a team’s budget and logs each call.
Common features include a single API across providers, fallbacks and routing rules, rate and spend limits, response caching (including semantic caching), input and output guardrails, and usage, latency and cost analytics that feed AI observability. Gateways for MCP tool servers apply the same idea to agents’ tool calls.
In episode 39 at 36:41, Vercel CTO Malte Ubl describes his company’s AI gateway: with it configured, developers can switch to any provider’s model by changing a string, without signing up for each provider’s keys (37:06). He is describing Vercel’s own product, and the episode was sponsored by Galileo. In episode 60 at 37:18, Redpanda CTO Tyler Akidau places LLM gateways and MCP gateways alongside identity and authorization as the controls enterprises need to govern agents, which Redpanda also sells.
A gateway is a single point of failure and of trust: if it is down or misconfigured, every AI feature is affected, and it sees every prompt. It also can’t make a weak prompt or the wrong model good; it only makes those choices visible and controllable. See least privilege for scoping what agents behind it may do.
Hear it from the guest
“You can use every AI model from any provider without any API keys.”
“So we're giving you the LLM gateways, the MCP gateways, the identity and authorization stuff that you need”
Quotes lightly edited to remove filler words.
Sources
- Cloudflare AI Gateway docs — Documents caching, rate limiting, spend limits, dynamic routing, guardrails, data loss prevention, logging and cost analytics for AI requests; checked October 24, 2026.
- Microsoft Learn: AI gateway capabilities in Azure API Management — Describes the AI gateway as capabilities to secure, scale, monitor and govern the models, agents and tools behind an application; checked October 24, 2026.
- Vercel AI Gateway docs — Routes requests across providers and fallback models and records status, provider, latency, token usage and cost for each attempt; checked October 24, 2026.
