Think of a support ticket that can be “new”, “in progress”, “waiting on customer” or “closed”. If the chance of a ticket moving from “in progress” to “closed” tomorrow depends only on where it is today, not on every step it took before, you can describe the whole system as a Markov chain. The table of move probabilities between every pair of states is called the transition matrix.
That “only the present matters” rule is the Markov property. It is a simplifying assumption, and real systems often break it, but it makes long-run questions answerable: how many tickets will be closed after a week, or which state the system settles into.
Why it matters for AI: Markov chains are the building block for the Markov decision process, which adds actions and rewards and underpins reinforcement learning. They are also a useful contrast with a large language model: a simple Markov text generator predicts the next word from only the last word or two, while an LLM conditions on everything in its context window. This is editorial background; no Chain of Thought episode has discussed Markov chains directly.
Sources
- Grinstead and Snell, Introduction to Probability (Chapter 11: Markov Chains) — Free textbook hosted by Dartmouth; Chapter 11 defines Markov chains, states and the transition matrix, checked October 24, 2026
- Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed.) — Authors' free edition; Chapter 3 states the Markov property used by reinforcement learning, checked October 24, 2026