Non-Determinism

A calculator is deterministic: 2 + 2 gives 4 every time. Most language-model products are not. Ask a chatbot to write a product description twice and you will usually get two different drafts. That is often what you want from a writing tool. It is a problem when a support bot answers a refund question correctly nine times out of ten and invents a policy the tenth time.

Some of the variation is built in. A model picks each next token by sampling from a list of likely options, and temperature controls how adventurous that sampling is. Turning temperature down reduces variation, but it does not remove it. Thinking Machines Lab found that hosted APIs and open-source inference engines still return different completions at temperature 0. Their explanation is that the serving software groups requests from many users into batches, and the math inside the model can change slightly with batch size, so your answer can depend on how busy the server is (Thinking Machines Lab). Outputs also change for reasons outside the model itself: a provider updates the model version, the retrieved documents change, or a small edit to the prompt shifts the answer.

This is one reason teams use AI evaluation. Deterministic code produces the same result for identical inputs under the same conditions, but one passing case still does not cover every input or path. With a nondeterministic system, even repeating that same case can produce a different result. Teams build eval sets of many realistic cases, rerun them, and track pass rates over time instead of checking a single output by eye. Where an answer must be exact, they add checks around the model, such as structured output schemas, validation code or a human approval step. A schema can check an answer’s shape without proving its factual accuracy.

On the show, guests describe this as the core engineering problem rather than a footnote. In episode 42 at 3:13, Aishwarya Srinivasan says the model’s nondeterministic nature makes edge cases hard to find, which pushes teams toward eval sets, and then asks how good those eval sets really are. Later in the same episode (49:42) she points to the Thinking Machines work as evidence that some variation comes from the “plumbing” around the model. In episode 44 at 12:09, Cisco’s Jeetu Patel calls it a “trust deficit”: companies want predictable applications built on a foundation that is unpredictable by definition. Those are the guests’ framings. The practical takeaway, which is our synthesis, is to test the same task many times and measure how often it succeeds before you rely on it.