AI Infrastructure
The compute, inference, and platforms underneath AI.
AI infrastructure is the compute, inference, networking, and data foundation that AI systems run on — from GPUs and model-serving platforms to the pipelines that move data through them at scale.
Agent infrastructure also has to handle work that arrives while a consumer is unavailable. For an event-triggered workflow, define the retention window, restart position, duplicate handling and owner of failed work. Microsoft's event-driven architecture guide distinguishes durable event streams from other delivery patterns and describes ordering, recovery and consistency tradeoffs. Webhooks covers the HTTP callback; a format or transport choice alone does not establish the recovery behavior your workflow needs.
Serving a language model includes managing the state of active requests. vLLM's Paged Attention documentation shows one concrete mechanism: splitting key and value cache data into fixed-size token blocks (vLLM, Paged Attention). That is a different concern from the model's answer quality. For your own service, measure the workload you expect to operate: prompt length, generated length, simultaneous requests and the time a user waits for a completed response.
Jeetu Patel in episode 44 discusses the networking and data-centre foundation of AI systems. Anush Elangovan in episode 72 moves down to profiling model workloads and tuning GPU kernels. How to cut AI agent costs follows cost through a full agent task, and why is it hard to move off NVIDIA? covers the software surrounding hardware choices. Follow AI energy and data centres for the power side of capacity planning.
Start here
- The Critical Infrastructure Behind the AI Boom | Cisco CPO Jeetu Patel
Follow the move from one GPU to racks and data centers that need coordinated networking.
- The First Fully Autonomous AI Attack is Coming | Kris Lovejoy, Kyndryl
See how data center density and existing equipment complicate a shift to GPU computing.
Go deeper
8 episodes
- For AI Agent Fleets, the Model Is Rarely the Bottleneck | Theory Ventures' Andy Triedman
- Tuning GPU Performance with AI Agents | AMD’s Anush Elangovan on ROCm 10
- The First Fully Autonomous AI Attack is Coming | Kris Lovejoy, Kyndryl
- Gemini 3 & Robot Dogs: Inside Google DeepMind's AI Experiments | Paige Bailey
- The Critical Infrastructure Behind the AI Boom | Cisco CPO Jeetu Patel
- Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture | Maxime Labonne
- AI's Two Extremes – Foundations & The Frontier | Databricks’ Denny Lee
- AI Infrastructure & the Evolution of RAG | Weaviate's Bob van Luijt
Explainers on this topic
- What a Context Window Is, and Why It Fills Up
- How Much Electricity AI Data Centers Use
- Why AI Runs on GPUs
- How to Reduce AI Response Latency
- What Is Data Lineage for AI?
- What a Token Is, and Why It Sets Your AI Bill
- When an Open Model Beats a Frontier API on Cost
- Centralizing Data Before AI: Why the Lake Doesn't Save You
- What AI Actually Costs the Environment
- The Switching Cost in AI Hardware Is Software
- Small Language Models in Production
Terms on this topic
- AI Chip Export Controls
- AI Energy Use
- AI Factory
- AI Gateway
- AI Interconnect
- Application-Specific Integrated Circuit (ASIC)
- Autotuning
- Batch Inference
- Compound AI Systems
- Compute
- Continuous Batching
- CUDA
- Data Center
- Data Lake
- Data Pipeline
- Data Sovereignty
- Distributed training
- Edge AI
- Event-driven architecture
- Feature Store
- FLOPs
- High-Bandwidth Memory (HBM)
- Hyperscaler
- Inference
- Inference Throughput
- Jevons Paradox
- KV Cache
- Mixture of Experts (MoE)
- Model pruning
- Model Routing
- Model Serving
- Neocloud
- On-Premises AI
- Prefill and Decode
- Prompt Caching
- Quantization
- Rate Limit
- Semantic Caching
- Sovereign AI
- Speculative Decoding
- Streaming Response
- Tensor
- Time to First Token
- Token Pricing
- Transformer
Tool comparisons on this topic
- Document parsers for RAG compared: choose how files become text
- LLM gateways compared: choose where model traffic is controlled
- Vector databases compared: choose by data, filters and operations
Guests on this topic
Andy TriedmanAnush ElangovanKris LovejoyPaige BaileyJeetu PatelMaxime LabonneDenny LeeBob van Luijt