AI Engineering
The discipline of building reliable AI products.
AI engineering is the discipline of building software powered by AI — choosing and adapting models, engineering prompts and context, evaluating and testing, and shipping LLM features that hold up in production. Distinct from using AI to write code (that is AI coding) and from the research of training models from scratch.
AI engineering starts with a task and a way to judge its output. Anthropic's engineering guide distinguishes the record of an agent's actions from the outcome those actions leave behind: a booking confirmation in chat is different from a reservation in the database. It describes code-based, model-based and human graders, each with different limits (Anthropic, Demystifying evals for AI agents). For a builder, that suggests a useful starting exercise: define the successful end state before choosing a model or framework.
This hub follows the work around that decision: inspecting failures, assembling context, testing changes and operating the result. In episode 37, Hamel Husain grounds evaluation in error analysis rather than a dashboard of generic metrics. Malte Ubl in episode 39 discusses moving agents from a prototype to production. Read how to test an AI system for a testing workflow and observability vs evaluation vs benchmarking to decide what each measurement is for.
5 episodes
- Explaining Eval Engineering | Galileo's Vikram Chatterji
- Architecting AI Agents: The Shift from Models to Systems | Aishwarya Srinivasan
- Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain
- Using AI to Modernize Your Legacy Applications | MongoDB’s Rachelle Palmer
- AI in 2025: Agents & The Rise of Evaluation-Driven Development | Galileo's Yash Sheth & Atindriyo Sanyal
Explainers on this topic
- How to Prepare Data for Fine-Tuning an LLM
- How to Reduce AI Response Latency
- The AI Hiring Doom Loop
- Code Review Is the New Bottleneck
- Small Language Models in Production
- How to Test an AI System
- When AI Hallucinations Are Good (and When They're Dangerous)
- 4 Things That Turn a Model Into an Agent
Terms on this topic
- A* search
- AI Engineer
- AI Observability
- AI Workflow
- AI-Native
- API (Application Programming Interface)
- Canary Deployment
- Chatbot
- Conversational AI
- Cost per PR
- Cost per Task
- Cross-Model Review
- Data Science
- Direct Preference Optimization (DPO)
- Eval-Driven Development
- Event-driven architecture
- Fine-Tuning
- Forward Deployed Engineer
- Heuristic
- Idempotency
- In-Context Learning
- Jupyter Notebook
- Knowledge Distillation
- LLM Tracing
- LLM Wrapper
- LLMOps
- LoRA (Low-Rank Adaptation)
- Machine Learning Engineer
- Max Tokens
- MLOps
- Model Registry
- Model Routing
- Non-Determinism
- Parameter-Efficient Fine-Tuning (PEFT)
- Prompt
- Prompt Chaining
- Prompt Management
- Regression Testing
- RLHF (Reinforcement Learning from Human Feedback)
- Role Prompting
- Self-Correction
- Shadow Deployment
- Structured Output
- Supervised Fine-Tuning
- Technical Debt
- Temperature
- Top-p and Top-k Sampling
- Training-Serving Skew
- Webhook
Tool comparisons on this topic
- AI agent SDKs compared: choose your orchestration layer
- LLM gateways compared: choose where model traffic is controlled
- LLM observability tools compared: choose where traces go
Essays on this topic
Guests on this topic
Vikram ChatterjiAishwarya SrinivasanHamel HusainRachelle PalmerYash ShethAtindriyo Sanyal