An online store wants a one-line summary of every review it has collected, 2 million of them. Nobody is waiting on any single summary, so the team submits them as a batch job overnight and loads the results into the product pages in the morning. The same store’s shopping assistant can’t work that way: a customer asking “does this jacket run small?” expects an answer in seconds, so that feature uses real-time inference.
The trade is speed for cost and simplicity. Batch jobs can wait for spare capacity, pack hardware efficiently and skip the always-on endpoint, so providers price them lower; Anthropic’s batch API, for example, costs half the standard rate. Real-time inference needs capacity ready for peaks and is judged on latency. Many systems mix both: precompute what can be predicted ahead (recommendations, embeddings for a vector database, nightly risk scores) and call the model live only for what can’t.
Batch work suits classification and tagging, summarizing archives, generating synthetic data, running large eval suites and backfilling fields in a database. It doesn’t suit chat, agents acting on a user’s behalf, or anything where the input only exists at request time. Batch results can also go stale: a score computed last night doesn’t know what happened this morning.
Don’t confuse batch inference with continuous batching, a scheduling technique inside a serving engine that groups live requests to raise throughput. See model serving for the online side and token pricing for how discounts show up on the bill.
Sources
- Claude Platform docs: Batch processing — The Message Batches API processes large volumes asynchronously for tasks that don't need immediate responses, at 50% lower cost; most batches finish within an hour, and batches expire if not complete within 24 hours; checked October 24, 2026.
- Amazon SageMaker AI docs: Batch transform — Recommends batch transform to get inferences from large datasets and to run inference when you don't need a persistent endpoint; checked October 24, 2026.
Go deeper
- Chip Huyen: Real-time machine learning, challenges and solutions article
Follow the move from batch prediction to online prediction with batch features, then online features, with the infrastructure each stage needs.