Rate Limit

A shared office printer that allows ten jobs a minute per person is a rate limit. AI APIs do the same thing to share capacity fairly and protect their systems. Limits are counted in several ways at once: Anthropic counts requests, input tokens and output tokens per minute, and Google counts requests per minute, tokens per minute and requests per day; going over any one of them triggers an error (Anthropic; Gemini API). The standard error is HTTP 429 Too Many Requests, which can include a Retry-After header saying how long to wait (RFC 6585).

Rate limits bite hardest with AI agents and batch jobs. One user question can turn into dozens of model calls, and a long document can use a minute’s token allowance in a single request. Limits usually grow with account tier and spend, which is why model launches often mention them. In episode 12 at 4:12, Google’s Logan Kilpatrick announces Gemini 2.0 Flash as “fully available for production use with higher rate limits.” Conor plugs Galileo, the show’s sponsor at the time, at 18:07. In episode 71 at 44:58, Slack’s Jaime DeLanghe describes plans to bring more of Slackbot’s data handling to third-party agents, a capability not yet available to them at the time of the conversation, so developers could build agents “without a lot of like extra work to manage rate limits” and context. This is a roadmap claim from Slack. Episode 71 credits Walrus Memory and Svix as Season 4 presenting sponsors in its opening read.

To handle limits well: retry 429s with exponential backoff and respect Retry-After, queue and spread bursty work, cut tokens per request with prompt caching or shorter context, and track usage against limits in AI observability dashboards. Providers also set rate limits to blunt abuse such as model denial of service.