What 81 Conversations Reveal About Agents in Production

Across 81 Chain of Thought episodes, guests describe the tests, permissions, economics and human decisions behind putting agents to work.

Executive summary

Enterprise ROI and adoption received substantive discussion in 36 of the 76 episodes with guests. The recurring question was what work an agent should be trusted to do.

Introduction

A convincing agent demo leaves several decisions unresolved: which mistakes matter, what the agent may change, and whether the completed work justifies the cost. Those are the decisions this report is meant to help engineering leaders make. The evidence comes from conversations with people building models, deploying agents, evaluating their behavior and managing the teams that use them.

The report covers conversations published from November 6, 2024, through November 1, 2026. I use guest accounts to identify decisions worth testing on your own systems, from model selection to who approves a release. Methods and limitations explains the selection rules and disclosures.

1. ROI and adoption 36 of 76
2. Reliability and failure modes 29 of 76
3. Evaluation and testing 29 of 76
4. Model economics 26 of 76
5. Coding agents 25 of 76
6. Memory and context 23 of 76
7. Security and governance 23 of 76
8. Human judgment 20 of 76
9. Hype vs reality 17 of 76
10. Infrastructure and energy 13 of 76

Themes overlap, so an episode can count toward several. Counts measure discussion, not agreement or adoption.

Counts are episodes out of 76.

ROI and adoption leads the ten overlapping themes, appearing in 36 of 76 episodes with guests.

The gap between demos and operations

A convincing answer still has to earn permission to become an action. Dan Klein, whose company Scaled Cognition builds its own models around controllable behavior, worries the model sounds right and is wrong. Patel keeps a human approving actions until the agent earns trust. Zhang wants to know what the spend bought (eps 54, 67, 64).

Reliability and failure modes received substantive discussion in 29 of 76 episodes; ROI and adoption in 36 of 76.

• 36 of 76conversations spent real time on ROI and adoption
• 29 of 76on reliability and failure modes

Klein described the gap directly: “the system will do something plausible, but not actually correct.” (Dan Klein, ep 54, 3:30) He distinguished a reasonable response from one that follows the behavior a builder intended. In his account, the ability to assemble plausible outputs makes a prototype compelling while leaving substantial work before an enterprise can ship it (ep 54, 9:02, 19:39).

the system will do something plausible, but not actually correct.

Siva Surendira, whose company Lyzr offers an agent framework with built-in guardrails, described enterprise concerns about erroneous actions, using an incorrectly approved refund as an example (ep 21). He argued that incorporating guardrails into the framework gave enterprises a viable route to production. His promise was organizational reach: “enforce the guardrails to the entire organization.” (Siva Surendira, ep 21, 10:48)

Klein questioned the control available through conventional prompting: “you don’t really know what you’re gonna get from a prompt.” (Dan Klein, ep 54, 20:44) His criticism sets up the case for Scaled Cognition’s own model. Surendira described controls around the model; Klein wanted more dependable behavior from the model itself (eps 21, 54). These are complementary parts of the deployment problem.

Treat every guardrail as risk reduction until someone shows you how it fails. Ask what happens when a request takes an unexpected path, and whether the control still applies. A team needs evidence about the limits of its controls before it can decide which actions to delegate.

Patel described trust as something to build through observing decisions in the intended environment. He recommended retaining human approval until the organization had enough experience with the agent’s recommendations to consider removing it, with the threshold varying by use case (ep 67). That leaves a consequential decision with the builder and customer: specify which actions can proceed automatically and what evidence would justify expanding that permission.

Jiaona Zhang challenged teams to connect their spending to a defined result: “how do you get visibility in terms of your spend versus the outcomes that you’re actually trying to drive?” (Jiaona Zhang, ep 64, 5:03) She named revenue and time allocation efficiency as possible objectives. Later, she criticized mandates that encourage employees to maximize AI usage without asking whether the activity is efficient (ep 64).

For Zhang, adoption begins with a workflow someone wants help with. She suggested identifying a time-consuming task, understanding it with the person doing it, and improving that workflow so the benefit becomes tangible (ep 64).

May Habib, whose company Writer offers a platform for generative AI applications and workflows, described a blocker that comes before measuring success: “When we say we align to business logic, folks need to agree on what that logic is” (May Habib, ep 1, 43:35). She pointed to marketing work requiring taste, where a customer might not have agreed on the brand voice or what good work looks like (ep 1, 44:04). Zhang asked teams to name the outcome; Habib exposed the work required to agree on its definition. Have the people who will accept the output resolve conflicting examples before turning those examples into evaluation criteria.

Sriram Palapudi described how ServiceNow measured value: “How many people could actually solve their problems on their own?” (Sriram Palapudi, ep 4, 32:00) He connected self-service resolution to measuring success and ROI on AI, alongside time savings and reduced cognitive burden from capabilities such as summarization. His account ties the value claim to the user’s task (ep 4).

Alex Klug, of HP, agreed with the panel that “ROI isn’t necessarily always just financially driven, right?” (Alex Klug, ep 4, 38:00) He included benefits that are harder to measure, such as freeing a team to focus on more important decisions. Zhang stressed visibility into spending and outcomes; Klug allowed for value whose financial return might appear later (eps 64, 4). Both wanted a named objective. They put different weight on when it becomes a financial return.

For a pilot, make that choice explicit. If the intended benefit is less cognitive burden, define how the team will assess it. If the objective is time saved, include time spent checking and correcting the agent’s work. Give the reliability review and the business review a shared task to examine.

What builders measure

An evaluation score should help a team decide what to fix. Hamel Husain’s complaint about generic dashboards was that they could leave builders unable to make that decision, even after installing an evaluation tool (ep 37). Evaluation and testing featured in 29 of 76 episodes. The measurements served different purposes: diagnosing product failures, comparing models, or deciding whether a system was ready for a task.

Husain, of Parlance Labs, challenged the default package of hallucination, toxicity and conciseness scores. His objection was their weak connection to the product being built: “They don’t necessarily correlate with actual failures in your AI product, and they don’t actually mean anything.” (Hamel Husain, ep 37, 4:51) He described teams plugging in a dashboard, treating evaluation as handled, then reaching a dead end because they still did not know which failures deserved attention (ep 37).

They don't necessarily correlate with actual failures in your AI product, and they don't actually mean anything.

• 29 of 76conversations discussed evaluation and testing
• 100traces: Hamel Husain's starting point for error analysis

His proposed starting point was concrete. Read traces, write notes about what went wrong, group those observations into error categories, and count them before choosing what to prioritize. Husain suggested reviewing at least 100 traces as a starting heuristic, then continuing until new traces stop teaching you something new (ep 37, 10:01).

Ask what decision a score drives. If it won’t change what you fix next, it’s decoration. Ask the team to show an example behind each failure category and explain how a proposed test would catch it.

Chip Huyen identified another problem with generic scores: the definition can change underneath the people using them. Discussing coherence and relevance metrics, she said, “But they’re actually very highly dependent on the models you use, on the prompt you use for the judge.” (Chip Huyen, ep 5, 28:00) She warned that the people maintaining a metric and the people consuming it may be different, and that changes to the judging prompt can make scores difficult to compare over time (ep 5).

Record the judge model and prompt alongside the score. When the judging setup changes, rerun saved examples through both versions before treating a score increase as a product improvement.

Vivienne Zhang, of NVIDIA, described why this work has to continue after release: “The queries we’re going to get are much, much more complex and involve a lot of different edge cases.” (Vivienne Zhang, ep 5, 26:00) In NVIDIA’s chip-design copilot, she described failures where the answer did not exist in the database. Other failures involved chunks being cut at the wrong places. The fixes she identified were improving the data and changing the chunking strategy (ep 5).

Zhang’s example gives error analysis a specific job. In that account, an unsuccessful answer called for work on the information available to the system. For a retrieval-based application, carry that distinction into the evaluation: check whether the needed information was available and retrieved, then assess the answer against it. Preserve the failing query and retrieved material so the next change can be tested against the same case.

João Moura, whose company CrewAI provides an agent framework and an enterprise offering, described a release problem that a test of the new feature would miss. The new behavior works, but “it breaks like something that it used to work before and now they have to roll back” (João Moura, ep 29, 36:19). He described teams reacting by checking prompts manually before deployment, then seeking help as that process became difficult to manage (ep 29, 35:52, 36:19). Keep successful tasks in the test set alongside failures. A release should demonstrate that the new capability works and that previously accepted behavior survives.

Huyen emphasized whether generated code runs and does the requested job efficiently; Zhang emphasized whether the model knows the proprietary codebase and available libraries well enough to choose an approach (ep 5, 35:00, 36:00). A passing functional test answers only part of Zhang’s question.

Alex Ratner, whose company Snorkel AI builds benchmarks with partners including Harvey, argued that the ability to use models and agents had “outpaced our ability to measure their capabilities.” (Alex Ratner, ep 57, 3:09) His response was to improve shared measurement alongside tests for individual applications (ep 57, 10:41, 20:19).

Ratner also explicitly defended public benchmarks against the objection that models can overfit to them: “the fact that people overdo and overfit to something does not diminish its value as a tool.” (Alex Ratner, ep 57, 20:19) He called for public benchmarks, private benchmarks and use-case-specific measurement together (ep 57).

Ratner defended shared tests as a way to inspect capabilities and direct further work. Husain challenged generic product metrics that fail to identify useful fixes (eps 57, 37). For a team choosing tests, those arguments address different decisions: which capabilities to investigate and which product failure to fix next.

Jeetu Patel, of Cisco, explained why application tests matter: “if a model does well on one eval, that doesn’t mean the model is going to do well in your environment.” (Jeetu Patel, ep 67, 14:15) He urged teams to build their own evaluations and connected those results to decisions about routing prompts to cheaper or more expensive models (ep 67).

if a model does well on one eval, that doesn't mean the model is going to do well in your environment.

Before the next model change, compare saved tasks from your application and use the failures, regressions and judging criteria to make the release decision.

The agent stack after the model

Richmond Alake, Angie Jones and Tyler Akidau each put production reliability beyond model selection: they emphasized what an agent remembers, which tools it calls, and whose permissions it uses (eps 53, 48, 60). Picking a model settles none of those decisions.

Memory and context featured in 23 of 76 episodes; security, identity and governance in 23 of 76.

• 23 of 76conversations discussed memory and context
• 23 of 76discussed security, identity and governance
• 22 of 76had a guest name MCP

Memory has an abstraction problem

Alake argued that memory deserves explicit design from the start. “Memory should be a first class primitive in your systems that you’re building.” (Richmond Alake, ep 53, 8:20) He connected that position to how information is represented, which attributes support recall, and how a system decides what to forget. His example used Oracle AI Database to combine several types of search (ep 53, 15:38).

the vast majority of things you'd ask an agent to do are not intelligence limited, they are context limited.

Alake’s example went beyond retaining a conversation. He described an assistant that records the steps it took to answer a query, along with the answer, as workflow memory for subsequent use (ep 53, 18:30). In his account, the engineering task includes storing experience in a form the agent can retrieve later.

Jerry Liu of LlamaIndex offered a tentative, simpler abstraction: “memory is basically just a form of I guess like persistent context.” (Jerry Liu, ep 61, 42:17) He included database entries, updates to SaaS tools, and an internal wiki made of files. Liu said his views were still forming and that he had limited experience building more complicated memory systems. He preferred simple files as a starting point for general users (ep 61).

Alake’s design agenda extends beyond Liu’s starting point: a persistent record still needs rules for retrieval, updates and forgetting (eps 53, 61). I take Liu’s caution seriously. Start with files you can read. Move toward Alake’s design when retrieval or forgetting breaks in a way you can point to.

Mikiko Chandrasekhar, of MongoDB, which she positioned as a memory store for agent applications, added a coordination problem. She described blackboard memory as a place where “you have agents kind of come together to post like partial solutions” (Mikiko Chandrasekhar, ep 33, 23:09). She distinguished that shared work from a skills library that preserves a successful pattern for reuse, and wanted tracing suited to both (ep 33, 23:32, 23:58). This extends the memory decision beyond what an individual agent remembers. When agents collaborate, specify which partial results others can read or revise, and preserve enough history to identify where a shared answer went wrong.

Sudhir Hasbe, of graph database company Neo4j, argued that giving an agent a collection of tables and expecting it to discover the relevant relationships would fail. He described using a semantic map to represent data relationships and the meanings of business terms before the agent reasons over them (ep 55, 6:35).

Before choosing a memory product, specify what must survive between tasks, how the next task will find it, and when an old entry should stop influencing an answer.

Tool access creates a review obligation

Angie Jones of Block described the appeal of connecting an incident report to GitHub: an agent could investigate the code and propose a fix while the human responds to the page. She presented that as an example of a workflow enabled by connecting systems, with the human reviewing the proposed change (ep 48, 14:54).

Jones said Block maintained an allow list of Model Context Protocol (MCP) servers: “But even if I built one, security needs to review it to make sure I didn’t miss something.” (Angie Jones, ep 48, 27:53) Security reviewed servers before admission, including those built internally. She stressed that an internal developer could miss a problem without intending anything malicious (ep 48).

But even if I built one, security needs to review it to make sure I didn't miss something.

Jones also described connecting internally built MCP services to the company’s identity provider through OAuth. In her account, that made authentication more familiar for employees who had struggled with API keys, tokens, and scopes (ep 48, 30:22).

Block handles three things separately in Jones’s account: which MCP servers are allowed, how employees log in to them, and which tools are flagged destructive. She described settings that require human permission before an agent uses a destructive tool (ep 48, 29:25). An allow list covers only the first decision.

MCP appears in guest speech in 22 of 76 episodes. That measures discussion of the protocol, not whether the guest deployed it or approved a particular server. Jones’s account supplies the operational detail a mention count cannot: the review required before a connector becomes available to employees.

1. OpenAI 41 of 76
2. Google (the company) 37 of 76
3. Anthropic / Claude 35 of 76
4. RAG 27 of 76
5. MCP 22 of 76
6. Claude Code 17 of 76
7. Gemini 17 of 76
8. GPU 16 of 76
9. Meta / Llama 16 of 76
10. Cursor 14 of 76
11. NVIDIA 11 of 76
12. GitHub Copilot 9 of 76

Episodes where a guest named the term at least once. A mention is not an endorsement or a deployment. MCP is the only protocol in the top 12.

Counts are episodes out of 76.

MCP is mentioned in 22 of 76 episodes with guests; counts reflect mentions by guests, not hosts.

Authority must follow the task

Tyler Akidau of Redpanda pushed the identity question down to individual agent instances and tasks. He argued against treating agents as conventional service accounts with a broad, standing set of capabilities (ep 60, 10:29).

agent safety and security is actually going to accelerate adoption because it's going to give people the confidence to be able to use these tools.

His proposed identity also preserves who delegated the work: “you need a hybrid identity that carries along with it this chain of responsibility” (Tyler Akidau, ep 60, 14:01). That chain could run from an agent through another agent back to the human on whose behalf they act.

you need a hybrid identity that carries along with it this chain of responsibility

Akidau paired identity with narrow permissions that expire. He also proposed explicit denials that remain in force when an agent acts for a more privileged human. In his example, an agent prohibited from writing to production should not acquire write access just because its user has it (ep 60, 17:06). He connected this design to revocation: a central governance layer could cut off an agent’s access when it needs to stop (ep 60, 30:10).

Charles Guillemet, whose company Ledger offers hardware devices for signing and managing crypto assets, set a stronger requirement for high-value assets: “the only way to have very strong security guarantees is to have like dedicated hardware.” (Charles Guillemet, ep 65, 34:49) He described hardware signatures for approving policies and isolating transaction keys, with proposed actions passing through a policy engine (ep 65, 9:24).

Guillemet scales the control to the asset. He conceded that software and access control could suffice for lower-value assets, then described stronger hardware controls as the value and required protection increased (ep 65, 34:49, 43:42). Jones’s allow list, Akidau’s delegated identity and Guillemet’s signatures address different parts of authorization. Choose the combination around the asset and the action an agent is allowed to take.

Maryam Ashoori, of IBM, which provides models and governance tooling, described a separate requirement from an insurance customer: “I need to know exactly what version of what model was trained on what version of data.” (Maryam Ashoori, ep 17, 16:54) In her account, the customer also needed to distinguish synthetic data from other training data and retain it for audit. This was her report of a customer’s requirement, not a universal rule. Permissions govern what the agent can do; model and data lineage help explain the system behind that action. Ask a supplier what records it can provide before committing to a model your organization must be able to audit.

Before expanding authority, test one complete task, including a prohibited write and revoking access during execution.

The economics of model choice

The strongest case for open weights in these conversations still leaves room for frontier models. Fergal Reid described Intercom replacing a query-preparation model while keeping Sonnet for its hardest answer prompts (ep 49). Joel Hron planned a similar division for legal work: a specialized model inside a product that also uses models from frontier providers (ep 69). Model choice becomes a decision about which model should do each part of the job.

Model economics received substantive discussion in 26 of 76 episodes, spanning provider access, control over model weights and smaller models for constrained tasks.

About $250K a month Intercom's recalled query-preparation inference bill, before specialization cut almost all of it Fergal Reid · ep 49, 6:55

About $40M Thomson Reuters' investment over several years in people, compute and methods Joel Hron · ep 69, 48:59

What the inference bill buys

Logan Kilpatrick of Google DeepMind framed the margin problem directly: “the more AI you add into a product experience, the higher your cost is, the lower your margins end up being” (Logan Kilpatrick, ep 12, 35:41). His context was making AI affordable enough to put in front of users. He presented cheaper Gemini models as a way to remove that barrier, while positioning Pro for harder coding and agent use cases (ep 12).

In the February 2026 episode, Reid recalled spending about $250,000 a month on summarizing and standardizing users’ queries before retrieval. He said Intercom post-trained an open-weight model for that task and saved almost all of that inference spend. The monthly estimate was from memory and covered query preparation (ep 49, 6:55).

The task matters. Reid explained that customers ask colloquial, inconsistent questions, and that rewriting those questions improved retrieval and the eventual answer. He said Intercom had long tried to run these supporting prompts on the smallest, cheapest, lowest-latency model that worked. The opportunity widened when “the open weight models started to become close to the closed weight models” (Fergal Reid, ep 49, 6:55). He qualified that comparison around Intercom’s own core tasks.

the open weight models started to become close to the closed weight models

Reid also described an internal AI-infrastructure team running distributed training on AWS H200 nodes (ep 49, 11:09, 11:25). Include the cost of that work when comparing a specialized model with a provider’s bill. The substitution decision needs a budget for operating the model as well as a test of its answers.

Reid’s case for specialization extended beyond price. For customer support, he wanted tighter control over when an answer should be escalated to a human. He described earlier fine-tuning experiments that failed to beat better prompting, followed by a later point when the task needed less additional general intelligence and more control over behavior (ep 49). His advice was conditional: “when your application matures, maybe when you no longer need frontier levels of intelligence” (Fergal Reid, ep 49, 2:27). That is a useful threshold to test before funding a training effort.

Reid said Intercom tested model changes in production and checked that explicit customer confirmations of resolution did not decline, rather than relying only on users going silent (ep 49).

What ownership buys

Joel Hron of Thomson Reuters put a longer-term value on controlling weights. His starting point was specialized legal knowledge and expert judgment: “we want to take a great open source foundation and add to it what our users really care about.” (Joel Hron, ep 69, 9:47) He described training around users’ jobs while trying to preserve the foundation model’s general capabilities.

Hron’s rent-versus-own analogy concerned what happens to expert feedback. He described attorneys comparing answers during evaluation, with their judgments informing changes to prompts, tools, and the agent harness. With control over the model, he wanted those judgments to become training data that improved its capabilities over successive versions (ep 69).

“If you rent a house, you’re paying a landlord and you’ve got a roof over your head and the landlord takes care of the house and makes sure everything works and that’s great and that works, but you build no long-term equity.” (Joel Hron, ep 69, 21:54)

For a team considering ownership, identify the expert decisions worth learning and evaluate whether training on them improves the target task. Track that improvement against the effort required to collect and use the feedback.

Hron also supplied a cost boundary: “We probably spent somewhere around $40 million in the last several years between people costs and compute and development of these methods and things like that.” (Joel Hron, ep 69, 48:59) His approximate multiyear investment covers a much broader scope than Reid’s monthly query-preparation bill. A team budgeting for ownership should separate those categories before estimating its own payback period.

Even Hron’s ownership strategy preserved a role for frontier providers. He said CoCounsel used Claude heavily alongside other provider models. At the time of the interview, Thomson’s first planned deployment inside CoCounsel was bulk document review, including due diligence. He envisaged it taking on further functions such as search, reranking, and citation analysis, with general frontier intelligence serving as an orchestrator (ep 69). The proposed specialization had a defined starting task.

Where small models fit

Maxime Labonne of Liquid AI described device deployment in terms of speed, memory and output quality: “usually there’s a trade off between the memory usage, the inference speed, and the quality.” (Maxime Labonne, ep 43, 3:38) He offered structured extraction into JSON as one small-model use, and also described small models processing data on GPUs at scale (ep 43). Model size, in his account, follows the job and its constraints. He drew a boundary around complex agent workflows: “It will not be able to replace cloud models quality if you need some kind of complex workflows.” (Maxime Labonne, ep 43, 15:05) Test a substitution on the step it will perform, including when it needs to hand work to a more capable model.

Require a substitution decision that includes both the serving bill and the effect on completed customer tasks.

Coding agents and the shape of engineering work

In Greg Foster’s account, faster code generation puts more pressure on the people deciding what ships: review and deployment become bottlenecks as agents produce more changes (ep 40). Anush Elangovan described a more aggressive response in his personal workflow: let tests govern the work and give agents room to implement it (ep 56). Their accounts place different weight on human understanding before accepting code.

Coding agents featured in 25 of 76 episodes; human judgment and organizational change in 20 of 76.

• 25 of 76conversations discussed coding agents
• 20 of 76discussed human judgment and organizational change

In April 2026, Elangovan, of AMD, described a radical approach to the development process: “In my personal space, I treat all code like ephemeral.” (Anush Elangovan, ep 56, 15:21) He described letting agents rewrite implementations and accepting reduced human readability when tests pass. His willingness to delegate extended to choosing Rust for projects despite saying he did not know the language (ep 56).

A year earlier, in April 2025, Charity Majors, of Honeycomb, had defined the job more broadly: “Software engineering is not about writing code. It is about solving business problems with technology” (Charity Majors, ep 18, 6:56). She emphasized maintaining and operating software, instrumenting it, and tracking down difficult bugs (ep 18). Her definition gives a way to assess Elangovan’s later workflow: does it preserve the team’s ability to own the resulting system?

If a team can specify the required behavior well enough to test it, Elangovan’s approach gives agents substantial freedom. Majors’ argument asks leaders to judge that freedom against the work of maintaining the system (eps 56, 18). I want evidence of both before using code output to justify a staffing decision.

The argument over review

Describing his personal workflow, Elangovan was explicit about acceptance: “testing is the code review and testing is kind of the gate of your sandbox” (Anush Elangovan, ep 56, 18:39). He described making performance requirements executable so an agent has to keep searching when a change fails them. He also described narrowing a task using his engineering experience and inspecting proposed tests before implementation (ep 56). The human work includes deciding what the agent must prove.

Foster, of Graphite, put more weight on what happens when another person reads a change. “You as an IC, you got to review more stuff. Your human time is still your human time.” (Greg Foster, ep 40, 18:13) He described a reviewer potentially becoming the first person to read code produced by a bot, with less confidence that its author had already thought it through (ep 40).

For Foster, review also carries information that a test suite may never have received. A teammate can explain a change in direction from yesterday’s meeting, identify a local convention, or share context about how the codebase is evolving. He warned that if engineers absorb context neither while writing code nor during review, the team can lose its understanding of the system (ep 40).

That is the substantive tension between Foster and Elangovan. Elangovan described encoding objections in the test harness. Foster described review as a place to discover objections and spread understanding. An engineering leader adopting the first approach still needs an answer to the second: who notices a requirement that has not yet become a test, and how does that knowledge reach the rest of the team?

Rachelle Palmer, of MongoDB, where she described using LLMs to modernize legacy applications, supplied a specific test of preservation: “you actually need to generate effectively equivalent tests between your legacy app and your new modernized app.” (Rachelle Palmer, ep 14, 8:39) She described using LLMs to help create those tests while retaining manual effort (ep 14, 9:05). Unlike a new feature with a fresh specification, a replacement has an existing system to compare against. Before accepting a rewrite, have the reviewer explain which old behaviors the tests preserve, which deliberately change, and which remain untested. Passing tests is more informative when the team can explain what they cover.

Foster’s proposed response was to keep changes small enough for meaningful review, using a stack of smaller changes so reviewers and tests can work at a finer level (ep 40). Track time waiting for review alongside time spent generating code. An unexplained approval should remain a reason to stop.

Product judgment becomes a larger part of the job

Laurie Voss combined a strong automation forecast with a specific limit. He argued that the coding craft centered on syntax and implementation was going away, while senior review and operations still required human involvement (ep 74). On deployment, architecture, and scaling, he said: “Those are definitely tricky, subtle areas where you need a lot of domain expertise and you need a lot of stuff that we have never written down.” (Laurie Voss, ep 74, 8:15)

it is a craft and the moment you forget that and just make it into you know, some type of balance sheet exercise … you lose a lot of the understanding of how systems come together.

Voss also argued that product sense was becoming central: “what to build in the first place is becoming absolutely central to the job.” (Laurie Voss, ep 74, 13:04) Later, he described a product engineer who translates a customer’s needs into software requirements and works with agents to implement them (ep 74). In that account, knowing how to interview a user and define useful behavior becomes a larger share of engineering work.

what to build in the first place is becoming absolutely central to the job.

Claire Vo, then LaunchDarkly’s chief product and technology officer (and founder of ChatPRD), brought urgency to that change. “I think velocity will be a massive, massive differentiator.” (Claire Vo, ep 36, 10:34) She described worrying about competitors building quickly with AI. But her account of adoption at LaunchDarkly depended on existing engineering practices that protected quality and on giving a trusted, experienced engineering leader responsibility for adoption (ep 36).

Vo said she chose someone who understood the architecture and had enough credibility to tell colleagues when a tool worked and when it did not (ep 36). Assign that responsibility explicitly, with time to investigate failures and improve how the team uses the tools.

Keep a path for people to learn

Majors resisted an organization built only around experienced hires. “This is an apprenticeship industry, full stop.” (Charity Majors, ep 18, 9:06) She described a junior engineer using ChatGPT to explore questions before bringing a better question to a senior colleague. She saw AI supporting that learning process and urged senior engineers to advocate for juniors (ep 18).

This is an apprenticeship industry, full stop.

Voss placed more emphasis on changing what people learn. He argued for education in the business of software and for training people to translate user needs into requirements (ep 74). Majors’ emphasis on apprenticeship and Voss’ emphasis on product judgment suggest a concrete training assignment: let a junior engineer help define a small change, use an agent to implement it, and explain the result with an experienced reviewer.

For the next agent-assisted change, name who owns the user problem and who accepts the implementation, then have a less experienced engineer help explain the tests.

Physical limits and the bubble question

Kelly Vaughn called agentic AI a bubble and still encouraged her team to use AI tools (ep 15). Jeetu Patel warned that physical infrastructure could not satisfy AI demand (ep 44). Those positions leave builders with a harder job than choosing optimism or skepticism: deciding which workloads justify the resources they consume.

Infrastructure, hardware and energy featured in 13 of 76 episodes; hype and bubble talk in 17 of 76.

• 13 of 76conversations discussed infrastructure, hardware or energy
• 17 of 76discussed hype, reality or a bubble

Capacity claims need a specific resource

Patel described a shortage across the physical stack: “There is just not enough power, compute, network bandwidth, and data center capacity to go satiate the needs of AI.” (Jeetu Patel, ep 44, 11:39) Cisco supplies infrastructure for those needs; Patel described its aim as helping customers generate tokens with less power and spending (ep 44, 15:24, 34:24).

He argued that power availability was determining where data centers were built, and that some training runs would require GPUs spread across multiple facilities operating as a coherent cluster (ep 44). His account puts a physical condition on promises of abundant intelligence: the infrastructure has to support the work.

Sharon Zhou, introduced as AMD’s vice president of AI in the episode, described another constraint on using available hardware: finding a working route through its software. AMD supplies the hardware and software ecosystem she was discussing. She warned about developers “frantically trying a bunch of things, just seeing documentation and getting a little intimidated and not knowing where to start even.” (Sharon Zhou, ep 28, 43:42) She wanted a prescribed path to a working result, while inviting community contributions. Patel’s capacity question and Zhou’s usability question both belong in procurement. Run the intended workload through the provider’s documented setup before treating available compute as usable capacity.

For a capacity plan, specify the workload, its expected concurrency and the latency users will tolerate. Ask the provider what capacity it can commit to in the required location. Use those commitments when deciding what volume of work to promise customers.

Water accounting changes the argument

Andy Masley challenged the scale of some environmental claims: “AI data centers just really don’t seem to be a significant issue for water in The United States.” (Andy Masley, ep 45, 3:22) His assessment concerned water use in a specific geography.

AI data centers just really don't seem to be a significant issue for water in The United States.

His example concerned a forecast for 2027. Masley said an account presented AI’s projected water consumption as about half of Britain’s water use. He argued that applying the water management definition of consumption to the cited study instead gave about 5% of Britain’s use. In his explanation, much of the water in the larger figure would be used at off-site power plants and returned to its source (ep 45).

Before using a water comparison in procurement, request the original calculation and check whether it measures withdrawal or consumption, for which geography and period.

A bubble can contain useful products

Vaughn’s bubble criticism focused on crowded product categories: “Like, how many AI notetakers seem to exist?” (Kelly Vaughn, ep 15, 1:51) She described fundraising cycles that draw startups toward fashionable categories and create a market of similar products (ep 15).

But her account also included established companies incorporating agents into their workflows because “they see the value in agentic AI and they can see a way to improve their customer’s life” (Kelly Vaughn, ep 15, 9:03). She also supported AI assistance with pull request review (ep 15).

She encouraged her team to adopt tools such as Cursor, while insisting engineers keep questioning the output (ep 15, 1:51, 22:17).

Yujian Tang sounded more expansive about the technology: “the technology is definitely transformational.” (Yujian Tang, ep 51, 5:36) Yet he also said the spending felt like a bubble and that a correction comparable to the dot-com crash would not surprise him (ep 51). Like Vaughn, he allowed technological usefulness and market excess to coexist.

Commit additional capacity only after the workload meets the agreed threshold for accepted work, with the provider’s resource calculations attached where relevant.

What changed in two years

The themes show what guests chose to talk about. The words they used show how the field moved. I counted a handful of terms across all 446,623 words of guest speech, grouped by half-year, and checked which labs and tools guests named in each period.

• 446,623words of guest speech analyzed
• 65%of all words spoken came from guests
• 3.2themes discussed in the average guest episode

Agents took over the vocabulary. Guests said “agent” or “agentic” about 30 times per 10,000 words in the second half of 2025 and 57 times in the second half of 2026. “Context” more than tripled over the same year, from 5.2 to 17.5, which matches the memory and context arguments earlier in this report.

Term	2024 H2	2025 H1	2025 H2	2026 H1	2026 H2
agent / agentic	33.7	29.6	29.8	42.9	57.2
context	8.6	5.8	5.2	12.5	17.5
eval / evaluation	17.6	6.7	11.4	3.4	3.5
RAG / retrieval	14.9	3.9	2.7	6.2	1.7
MCP	0.0	1.8	1.6	5.8	2.3
open source / open weight	2.4	4.9	5.7	3.7	8.6
fine-tuning	1.6	2.7	2.7	1.4	0.3
guardrails	4.3	1.5	0.9	0.5	1.0
Episodes with guests	7	18	16	18	17

Matches in named guest speech only. 2024 H2 covers seven episodes, so treat its values as rough. Full table: vocabulary-by-half-year.csv.

Two terms faded. RAG dropped from 14.9 uses per 10,000 words in the first episodes to 1.7, and from 86% of guest episodes in late 2024 to 6% in late 2026. The word “eval” fell too, from 17.6 to 3.5, even though evaluation is still one of the most discussed themes, with substantive discussion in five of the seven newest episodes. Part of the early figure is the show’s history: it launched with Galileo, an evaluation company, and its first episodes leaned on the subject.

The labs guests name shifted as well. Anthropic or Claude came up in 14% of guest episodes in late 2024 and 78% in the first half of 2026. OpenAI rose more slowly, from 29% to 61%. MCP, absent before 2025, came up in more than a third of guest episodes in 2026.

Term	2024 H2	2025 H1	2025 H2	2026 H1	2026 H2
Anthropic / Claude	14%	17%	31%	78%	71%
OpenAI	29%	50%	56%	61%	59%
Google / Gemini	29%	44%	62%	56%	65%
MCP	0%	28%	25%	39%	35%
RAG	86%	44%	31%	39%	6%
Meta / Llama	29%	28%	12%	33%	6%
Episodes with guests	7	18	16	18	17

An episode counts once per term, however often it was named. Google / Gemini combines the company and its models. Full table: mentions-by-half-year.csv.

Themes rarely travel alone. The average guest episode had substantive discussion of 3.2 of the ten themes, and the most common pairings join payoff to proof: evaluation and ROI came up together in 17 conversations, reliability and ROI in 17, and evaluation and reliability in 16.

1. Evaluation + ROI 17 of 76
2. Reliability + ROI 17 of 76
3. Evaluation + reliability 16 of 76
4. Model economics + ROI 15 of 76
5. Evaluation + model economics 14 of 76
6. ROI + human judgment 14 of 76

Episodes where both themes had substantive discussion. Full table of all 45 pairs: theme-pairs.csv.

Counts are episodes out of 76.

These are counts from one show’s guests, selected for a podcast. The early periods are small, so read the direction of a change as the finding rather than its exact size. The tables behind this section are downloadable: vocabulary, mentions and theme pairs.

Numbers guests cited

A selection of figures guests gave about their own work, each linked to the moment they said it.

Figure	What the guest said it measures	Source
~$250K/mo	Intercom's spend on one summarization task before it moved to open-weight models. Reid said he didn't remember the exact amount.	Fergal Reidep 49, 6:55
~$40M	Thomson Reuters' spend over several years on people, compute and methods for its legal model	Joel Hronep 69, 48:59
100 traces	A starting point for error analysis: keep reading traces until you stop learning anything new	Hamel Husainep 37, 10:01
60-65% → 90%+	Accuracy he described for pure RAG or vector search, versus customers combining graph and vector retrieval	Sudhir Hasbeep 55, 12:09
90,000+	Agents Cisco rolled out to its employees	DJ Sampathep 80, 10:53
120,000	Login attempts an agent swarm made in the example he described	DJ Sampathep 80, 37:37
6.5B tokens	Tokens his agent-first workflow had consumed when he checked that morning, up from the 4.2 billion he cited earlier	Anush Elangovanep 56, 10:54
500+ PRs	Pull requests one engineer produced in a short period; the low cost per PR hid very different work	Andrea Malagodiep 76, 4:33
Up to 30%	Savings on token spend from running some work on cheaper CPU inference	Andrea Malagodiep 76, 42:51
~40%	Candidates who walked away from a hiring process because it was so janky	Daniel Chaitep 58, 22:13
3-4 months	How often a new model might now arrive, against 8 to 18 months before	Jeetu Patelep 67, 11:38
Negative	Average time to exploit a vulnerability, a statistic he set out to explain	Dan Lorencep 68, 3:42

What guests predicted

Michel Tricot expected context engineering to be a transitional role lasting one to three years; Laurie Voss expected agents to struggle with unwritten knowledge for another five (eps 52, 74). Their forecasts concern different tasks. Keeping each task and its conditions attached makes the prediction useful for planning.

Better models, changing costs

On November 20, 2024, Sara Hooker, then at Cohere, forecast broader optimization in 2025, including data selection, algorithms and tooling (ep 3). She predicted diverging model sizes: “I think models are going to get bigger … But I also think models are going to get a lot smaller” (Sara Hooker, ep 3, 36:00). She associated larger models with multimodal work and smaller models with better optimization. Her forecast allowed for growth in opposite directions, depending on the task.

On August 12, 2026, Cisco’s Jeetu Patel described a possible model release cadence of three to four months. He expected a model’s time at the top to be short, with older models retaining uses because of their cost. He also warned that cheaper tokens could be offset by agents consuming more of them, and argued for reducing cost per outcome (ep 67). For planning purposes, keep the release cadence separate from the forecast about useful service life: Patel left room for an older model to remain worth running.

Which work agents would absorb

On March 25, 2026, Airbyte’s Michel Tricot put a horizon on context engineering as a distinct role: “I honestly think that it’s more a role for the next one, two or three years” (Michel Tricot, ep 52, 10:38). He expected improving and enriching context to become mechanical and self-learning, with agents eventually doing that work if development went according to plan. In the same episode, he predicted that existing ways of implementing retrieval-augmented generation would give way to different practices, while retrieval itself would remain useful. His example involved combining metadata with unstructured content to improve search (ep 52).

Voss offered a more cautious forecast on September 22, 2026. Discussing domain expertise and information people had never written down, he said agents would struggle: “they’ll continue to be bad at it for the next five years.” (Laurie Voss, ep 74, 8:15) He expected substantial retooling for programmers over that period, while rejecting overnight disappearance of every programming job (ep 74).

Tricot described shaping available data; Voss emphasized knowledge that had never been recorded (eps 52, 74). A staffing plan should distinguish those tasks before assigning either forecast to the whole engineering team.

Eiso Kant, whose company Poolside builds models for software development, offered a faster forecast on May 14, 2025: “It’s likely less than three years where you truly have human level capabilities in software development” (Eiso Kant, ep 23, 23:32). He included collaboration inside organizations and consequential software environments, with agents learning from company data and shared histories of their actions (ep 23, 23:56, 24:22). Kant’s broad capability forecast is more optimistic than Voss’s later warning about tacit expertise. Their forecasts leave a specific question to test: can an agent learn a team’s unwritten requirements well enough to operate its software? Do not assume a coding benchmark settles that question.

Product design and adoption

On February 26, 2026, Intercom’s Fergal Reid argued that customer conversations would favor “one agent or one set of agents from one vendor” (Fergal Reid, ep 49, 35:58). He anticipated conflicts and awkward handoffs between separate sales and service agents. That forecast explained Intercom’s investment in a customer agent spanning those roles; he gave no adoption deadline (ep 49).

Tormod Ree of Neat expected rapid adoption of agents for device fleet monitoring, configuration and repairs in his August 27, 2026 episode: “it’s going to be an agent that sort of then does actions to resolve issues, both directly in our platform, but also across platforms.” (Tormod Ree, ep 70, 24:05) He placed that development in the next year (ep 70).

Charles Guillemet of Ledger made adoption conditional on July 1, 2026. Discussing agents handling blockchain transactions, he first required strong safety and security guarantees, then predicted: “And adoption will be very fast.” (Charles Guillemet, ep 65, 31:39) He supplied no calendar deadline. Later, he also anticipated difficult months and years of security problems (ep 65). Removing the security condition would change his forecast.

Maturity and market corrections

On September 22, 2026, Voss forecast a market correction: “valuations are going to collapse.” (Laurie Voss, ep 74, 20:07) He expected disappointment followed by a slower recovery toward useful applications. He supplied no date for the correction and framed it as his expectation based on prior technology bubbles (ep 74).

valuations are going to collapse.

Before putting a forecast into the roadmap, record its task, timing and dependencies, then test the dependency your team would rely on.

Methodology

This report examines Chain of Thought episodes published from November 6, 2024, through November 1, 2026. Early episodes had unlabeled transcripts and were speaker-labeled from the audio before analysis; the words were unchanged. Attributable guest speech is available in 76 of 81 episodes.

The unit of analysis is an episode; some have several guests. Theme counts come from curated lists requiring an extended argument or repeated discussion by a named guest. They use the 76 episodes with guests as their denominator. Themes overlap, so totals cannot be added. These selections measure discussion within these conversations, not agreement or adoption.

Quotes and paraphrases use identified guest speech. Unidentified passages are excluded. The mechanical corpus excludes Conor Bronsdon, Vikram Chatterji, Atindriyo Sanyal and Yash Sheth as hosts. The Dev Interrupted crossover (ep 13) is counted as host-only because it has no Chain of Thought guest: Dev Interrupted’s Andrew Zigler hosts it and interviews Chatterji. Roles and affiliations follow episode metadata or introductions. Forecast dates are publication dates; timing and conditions stay attached to each prediction.

Attributable guest speech is available in 76 episodes across nine quarters; 81 published, 5 host-only.

The mentions chart counts each episode once per term, using named guest speech and a denominator of 76. Mentions indicate neither endorsement nor deployment. The chart separates Google company mentions from Gemini and Gemma model mentions; categories can overlap. Keyword matching depends on wording and transcription, so these counts cannot establish that a topic was absent.

Guests and topics were selected for a podcast, not randomly sampled. Guest-reported savings, results and forecasts are not independently audited; transcript verification checks wording, not audio or underlying claims. Separate interviews are not comparative tests. Practical recommendations are my editorial judgments. Many guests sell a product tied to their position; relevant vendor interests are disclosed when those guests first appear.

Vocabulary rates count case-insensitive matches in named guest speech per 10,000 guest words, grouped by calendar half-year. Mention shares count each guest episode once per term. Theme pairs count episodes that appear on both curated theme lists.

Conclusion: what to do Monday

Start with one workflow your team already understands. Name the user, the task and the expected benefit. Decide who accepts the result and who can stop the agent. Keep those names beside the criteria for expanding the deployment, so a successful demo has a clear next test.

Give the pilot owner a failure set built from saved traces. Preserve examples of mistakes and record what changed between runs. If an LLM judges the output, retain the judge model and prompt. Ask a domain expert to examine cases that passing tests cannot settle.

Test permissions on the same workflow. Attempt a prohibited write, try an unapproved connector, and revoke access during execution. Have the reviewer explain which control stopped each action and which failure still requires a person. That explanation belongs in the release decision alongside the successful runs.

Then compare the cost of accepted work with the quality threshold fixed. Include training, operations and human correction in a model substitution. Check the capacity required for the proposed volume. A lower serving bill deserves approval when the completed customer task still meets the requirement.

Make learning part of acceptance. Involve a less experienced engineer in explaining the tests and the user problem, then compare the released behavior with the requirement. Use the gaps to improve the next task’s instructions and review.

If no one can explain why the agent’s result is acceptable, keep its next action behind human approval. Give the person making that decision the failures and cost of completed work, so expanding the agent’s authority is a decision they can defend.