What is context in an AI agent?

In detail

At 8:12 in the interview, Jerry Liu describes the context layer as the services and information surrounding the model. The five categories below organize the examples in his answer. They overlap: Salesforce can be a connected tool, a source of structured data, and a system of record.

1. Documents

PDFs, slide decks, and other files contain information an agent may need to extract before it can use it. Liu points to document interpretation as LlamaIndex’s focus. The relevant question is whether the extracted text, tables, and relationships preserve what the task requires.

2. The web

Search and crawling can supply information beyond a model’s training data. Access to the web does not establish that a result is accurate or current; the source still needs checking.

3. Structured data

Database records and query results can ground an answer in specific business information. Decide which records and operations the task requires, along with the permissions to access them.

Update, October 3, 2026: WisdomAI’s approach gives shared business definitions an owner and a maintenance loop. In episode 75, Kapil Chhabra calls changing definitions “context drift.” Chhabra describes showing evidence of a change to a subject-matter expert, who decides whether the shared definition should change. At 34:04, Chhabra distinguishes institutional context from personal memory: the shared definition needs an authority who approves it. Episode 75 was sponsored by WisdomAI (sponsor thanks at 4:13).

4. Connected tools

Tools expose software capabilities to the agent. Liu gives MCP connections to services such as Confluence and Salesforce as examples. Reading information and taking an action require separate decisions about what the agent should be allowed to do. The MCP specification separates resources, context and data for the user or model to use, from tools, functions the model can execute. That is not the same line as reading versus acting: a tool can also just read, and MCP’s readOnlyHint annotation exists to mark such tools, so the permission decision still has to be made tool by tool. How AI agents use tools covers how the model chooses and calls them.

Access is part of the context decision. In episode 71 at 6:42, Slack CPO Jaime DeLanghe said Slack gives users, not itself, the choice of where an agent gets context: an agent working in a channel has access to that channel and, depending on how it is installed, could reach all public channels in the workspace. Episode 71 credits Walrus Memory and Svix as Season 4 presenting sponsors in its opening read.

5. Systems of record

An agent may need to store and later retrieve state in a database, a business application, or a collection of files. Liu returns to this idea when discussing memory as persistent context at 42:17.

Use the categories to investigate a failure

A stale policy document can produce a bad recommendation even when the model follows its instructions. Check the evidence and tool results the agent actually received before deciding whether to change the model, the instructions, or the retrieval system. This is a diagnostic example, not a claim that context explains most agent failures.

For a hands-on exercise, repair a stale agent handoff: check the changed decision, update the saved context, and inspect what a fresh session proposes next.