What is context in an AI agent?

The 5 Sources of Context an AI Agent Needs

Context is the information and capabilities supplied around a model so it can carry out a task. Jerry Liu describes documents, web information, structured data, connected tools, and systems of record as parts of that surrounding layer. These categories overlap; they are a way to inspect an agent’s inputs and access, not a universal architecture.

At 8:12 in the interview, Jerry Liu describes the context layer as the services and information surrounding the model. The five categories below organize the examples in his answer. They overlap: Salesforce can be a connected tool, a source of structured data, and a system of record.

1. Documents

PDFs, slide decks, and other files contain information an agent may need to extract before it can use it. Liu points to document interpretation as LlamaIndex’s focus. The relevant question is whether the extracted text, tables, and relationships preserve what the task requires.

2. The web

Search and crawling can supply information beyond a model’s training data. Access to the web does not establish that a result is accurate or current; the source still needs checking.

3. Structured data

Database records and query results can ground an answer in specific business information. Decide which records and operations the task requires, along with the permissions to access them.

4. Connected tools

Tools expose software capabilities to the agent. Liu gives MCP connections to services such as Confluence and Salesforce as examples. Reading information and taking an action require separate decisions about what the agent should be allowed to do.

5. Systems of record

An agent may need to store and later retrieve state in a database, a business application, or a collection of files. Liu returns to this idea when discussing memory as persistent context at 42:17.

Use the categories to investigate a failure

A stale policy document can produce a bad recommendation even when the model follows its instructions. Check the evidence and tool results the agent actually received before deciding whether to change the model, the instructions, or the retrieval system. This is a diagnostic example, not a claim that context explains most agent failures.

For a hands-on exercise, repair a stale agent handoff: check the changed decision, update the saved context, and inspect what a fresh session proposes next.