{ "schemaVersion": 1, "slug": "81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt", "episode": 81, "season": 4, "guid": "0a2a150e-f47f-44cf-b53d-3654a8f2941c", "title": "Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt", "canonicalUrl": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/", "publishedAt": "2026-11-01", "durationSeconds": 3427, "guests": [ { "name": "Saahil Jain", "company": null } ], "audioUrl": "https://media.transistor.fm/acd5ed31/e8bc1736.mp3", "transistorUrl": "https://share.transistor.fm/s/acd5ed31", "youtubeUrl": null, "summary": "Saahil Jain joined You.com as a founding engineer in 2020, when it was a consumer search engine competing with Google. Now its CTO, he builds search for AI agents, and he argues that however smart models get, they will still need tools to find out what is happening in the world. His analogy: we built the self-driving cars before the roads. He walks through which work he expects to move inside the model, why better models made harnesses matter less, and why cheaper open models and routing are giving harnesses a new job. He says he changed his mind on that last point in the past few months. Then we get into how agents search differently than people do: thousands of queries at once, recall over ranking, and a much wider range of latency needs. Saahil traces a query from fan-out to the index to re-ranking, and explains why You.com judges search by whether the agent answered the question, and why he trusts customer evals and benchmarks that change over time more than static public ones. We cover: - Core intelligence versus augmented intelligence, and the self-driving cars that arrived before the roads - Which tasks Saahil expects models to absorb, and where unverifiable rewards slow them down - Why frontier labs keep going vertical while open-model labs give weights away - How stronger models shrank the gap between good and bad harnesses, and how routing across open models can revive them - How You.com post-trains models with its search tool while trying not to overfit to any one model - How agent search differs from human search: query volume, context, ranking and latency - The life of a query: subqueries, lexical and semantic indexes, re-ranking and enrichment - Why static benchmarks can be memorized, and what dynamic benchmarks and customer evals measure instead Mentioned in this episode: - I Paid an AI Agent $8 to Write About its ‘Life’: https://newsletter.chainofthought.show/p/i-paid-an-ai-agent-8-to-write-about - MotionBench: https://motionbench.sh Connect with Saahil Jain: - LinkedIn: https://www.linkedin.com/in/saahiljain/ - Twitter/X: https://x.com/saahil9jain - You.com: https://you.com - You.com API docs: https://you.com/docs/quickstart Connect with Chain of Thought host Conor Bronsdon: Newsletter: https://newsletter.chainofthought.show/ Twitter/X: https://x.com/ConorBronsdon LinkedIn: https://www.linkedin.com/in/conorbronsdon/ YouTube: https://www.youtube.com/@ConorBronsdon More episodes: https://chainofthought.show Chapters: (0:00) Agents run a thousand searches at once (0:25) What survives as AI models get smarter (1:47) Is this the singularity? (4:26) Core intelligence versus augmented intelligence (6:33) Self-driving cars before the roads (7:27) What belongs inside the model (11:00) Where verifiable rewards run out (12:16) Can agents form relationships? (16:58) What model labs own, and what independents can build (20:21) What a harness is (22:39) The model is eating the harness (23:14) Open models and routing revive the harness (25:32) Open versus closed: what the labs keep (31:30) OpenAI, Baseten and the enterprise model budget (33:53) Co-evolving a search tool with the model labs (37:41) How agent search differs from human search (42:42) Life of a query, from fan-out to index (47:22) Measuring retrieval by whether the agent answered (50:21) Static benchmarks, freshness and customer evals (54:19) Why he changed his mind on harnesses Thanks to Walrus, presenting sponsor of season four of Chain of Thought. Walrus Memory gives AI agents portable, verifiable memory that carries context across apps, sessions, and other agents. Get started at https://walrus.xyz/cot Thanks to CoralBricks for sponsoring this episode. CoralBricks serves open models through one API, fast and at low cost: DeepSeek V4.1 Flash streams at over 300 tokens a second. Get $5 in credits on signup, plus another $5 on your first purchase: https://www.coralbricks.ai/r/Z6V2RE Thanks to Inngest, presenting sponsor of season four of Chain of Thought. Agents in production run long: they call models and wait on APIs and people. But the longer agents run, the more they break. Inngest handles that with durable execution. You build your agent as steps in TypeScript, Python, or Go. When a step fails, Inngest retries it with exponential backoff, and completed steps are saved and skipped. Try it out: https://inngest.link/cot-pod Thanks to WisdomAI for sponsoring this episode. In WisdomAI's latest benchmark, Claude Code alone topped out at 73% accuracy; WisdomAI closed the gap, delivering accurate answers 3x faster at about a third of the cost. Get \"The Blueprint for Trusted AI Analytics\": https://wisdom.ai/benchmark", "chapters": [ { "time": "0:00", "seconds": 0, "label": "Agents run a thousand searches at once" }, { "time": "0:25", "seconds": 25, "label": "What survives as AI models get smarter" }, { "time": "1:47", "seconds": 107, "label": "Is this the singularity?" }, { "time": "4:26", "seconds": 266, "label": "Core intelligence versus augmented intelligence" }, { "time": "6:33", "seconds": 393, "label": "Self-driving cars before the roads" }, { "time": "7:27", "seconds": 447, "label": "What belongs inside the model" }, { "time": "11:00", "seconds": 660, "label": "Where verifiable rewards run out" }, { "time": "12:16", "seconds": 736, "label": "Can agents form relationships?" }, { "time": "16:58", "seconds": 1018, "label": "What model labs own, and what independents can build" }, { "time": "20:21", "seconds": 1221, "label": "What a harness is" }, { "time": "22:39", "seconds": 1359, "label": "The model is eating the harness" }, { "time": "23:14", "seconds": 1394, "label": "Open models and routing revive the harness" }, { "time": "25:32", "seconds": 1532, "label": "Open versus closed: what the labs keep" }, { "time": "31:30", "seconds": 1890, "label": "OpenAI, Baseten and the enterprise model budget" }, { "time": "33:53", "seconds": 2033, "label": "Co-evolving a search tool with the model labs" }, { "time": "37:41", "seconds": 2261, "label": "How agent search differs from human search" }, { "time": "42:42", "seconds": 2562, "label": "Life of a query, from fan-out to index" }, { "time": "47:22", "seconds": 2842, "label": "Measuring retrieval by whether the agent answered" }, { "time": "50:21", "seconds": 3021, "label": "Static benchmarks, freshness and customer evals" }, { "time": "54:19", "seconds": 3259, "label": "Why he changed his mind on harnesses" } ], "hasTranscript": true, "segmentCount": 137, "transcript": [ { "index": 0, "time": "0:00", "seconds": 0, "speaker": "Saahil Jain", "text": "They can do a thousand searches at once, or even a hundred, you know, or even ten thousand searches at once. As a human, I've never seen a human. who's able to hit, 1000 QPS. but I have seen agents that are able to hit really high QPSs. when they're doing searches. Recall generally seems to matter more than precision when it comes to agent results, One thing for sure is that agents will be performing more searches than humans for the foreseeable future.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-0" }, { "index": 1, "time": "0:25", "seconds": 25, "speaker": "Conor Bronsdon", "text": "Welcome to Chain of Thought, everyone. With me today is Saahil Jain, CTO of You.com. We are here to talk about what survives as AI models become rapidly more intelligent. and the future of search. Saahil is great to see you. How are you feeling about the current state of AI advancement?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-1" }, { "index": 2, "time": "0:42", "seconds": 42, "speaker": "Saahil Jain", "text": "Hey, Conor. Excited to be here. Thanks for having me on. I think in terms of the future of AI development. Um there's there's a lot happening right now, so I think things are really accelerating fast. There's many different trends that we'll we'll talk about soon. But I think at a very high level, Right now we're seeing Yeah. two big trends. One is the model's capability is getting rapidly better. We're seeing a lot of competition amongst the frontier and open source. ecosystems and along with that rapid improvements and capabilities specifically on long horizon tasks.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-2" }, { "index": 3, "time": "1:13", "seconds": 73, "speaker": "Saahil Jain", "text": "So we're seeing AI models powering agents that are able to do more and more effective long-running tasks. whether it's in coding as well as domains outside of coding. The second trend that we see is open source models becoming Much better. So we see this from a variety of labs across the world. Uh-huh. working on these open source models that are Mm-hmm. They do lag behind the frontier models, but they're rapidly closing on certain types of tasks. So those are kind of the two mega trends that we see right now when it comes to AI development.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-3" }, { "index": 4, "time": "1:41", "seconds": 101, "speaker": "Saahil Jain", "text": "And both of these trends have a lot of implications in terms of how we build the product and how we build software right now.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-4" }, { "index": 5, "time": "1:48", "seconds": 108, "speaker": "Conor Bronsdon", "text": "The esoteric question this brings to mind for me is Uh depending on what side of Twitter you're on, you may be hearing that the singularity is beginning. Do you feel that's happening, or is that simply an overblown? Uh Mm-hmm. pro-technology discussions occurring.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-5" }, { "index": 6, "time": "2:06", "seconds": 126, "speaker": "Saahil Jain", "text": "Yeah. I I think uh I I think a lot of it depends on how one defines singularity, but I think for me I do think there is something Um unique happening right now compared to a lot of the technological revolutions in the past. I think this is fundamentally different than a lot of the previous ones. And the implications will be Mm-hmm. widely felt. Um in terms of how we work. Yeah, the knowledge economy as well as even the physical economy. So I think everything is rapidly changing.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-6" }, { "index": 7, "time": "2:35", "seconds": 155, "speaker": "Saahil Jain", "text": "As for singularity or not. I I don't know if I have a good uh sense of that yet. Um but I do think that we're in for a world of change. over the next couple of years and it's gonna require a lot of adjustment on everybody's part.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-7" }, { "index": 8, "time": "2:48", "seconds": 168, "speaker": "Conor Bronsdon", "text": "What is making today's rapid technology transformation from AI so different from past transformations we've experienced.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-8" }, { "index": 9, "time": "2:57", "seconds": 177, "speaker": "Saahil Jain", "text": "Yeah, and this is just my opinion. I'm sure there's many different takes here. But I think one of the the key pieces of it is there are similarities and there are differences and this is true of any you know, technological revolution. Um I think one of the key differences between what with what's happening now is The degree of intelligence. um of what's happening and the degree of autonomy. So I think what we're seeing is you know, level of autonomy. Yeah, that previous systems we're not capable of.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-9" }, { "index": 10, "time": "3:25", "seconds": 205, "speaker": "Saahil Jain", "text": "So that's one difference is the autonomy level is much higher than it was. from a lot of the developments in the past. when you think of the Industrial Revolution, et cetera. Um to a degree, right? So that's more of a scale. So I wouldn't say that's qualitatively different. But that is is one one area. But the second one that is probably the bigger difference is We now have AI that you know, is in many different domains, so across many domains. you know, often smarter than a human.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-10" }, { "index": 11, "time": "3:50", "seconds": 230, "speaker": "Saahil Jain", "text": "On what tasks that we would consider to be primary key knowledge tasks. So I think those are two. differences from previous revolutions. I don't, you know, I'm not necessarily saying these are completely qualitatively different. What I think quantitatively. Um it's it's very uh Yeah, I think there there are various differences and there's some similarities too. So it's not the first time that we've had. a tool that's smarter than a human, you know, a calculator is better at math than a human. Um but you know the way in which and the degree to which these tools can perform behavior. um that are aligned or in some cases misaligned with humans is quite different than Um the past.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-11" }, { "index": 12, "time": "4:27", "seconds": 267, "speaker": "Conor Bronsdon", "text": "And I think this gets to the core of something I have heard you talk about. a couple of times, which is the distinction between core and augmented intelligence and how that changes the way We decide what to spend our time on and what to build. Can you unpack that a bit for us?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-12" }, { "index": 13, "time": "4:43", "seconds": 283, "speaker": "Saahil Jain", "text": "Yeah, so this is how a little bit at You.com, how we've been thinking about the future. In general, we think of there being two different you know, trends when it comes to horizontal AI. So when I say horizontal AI, we're talking about AI that applies across. Disciplines? So you can think that is generalist AI improvements. In our head, you know, we view the world in two buckets. One bucket is core intelligence. So core intelligence are the underlying foundation models. that are getting better week after week.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-13" }, { "index": 14, "time": "5:09", "seconds": 309, "speaker": "Saahil Jain", "text": "And this is what I refer to. When I discussed, you know. model capability is improving rapidly. In both the open source and frontier domains. So that's the first piece about core intelligence, the underlying foundation models. You can think of it as the engine behind a lot of the growth. in the AI economy. And then when we think about augmented intelligence, These models are super capable. Um but they need tools and services in order to fulfill their jobs the same way we as humans are very smart, but we still need to use tools.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-14" }, { "index": 15, "time": "5:36", "seconds": 336, "speaker": "Saahil Jain", "text": "The tool that we focus on is AI Search. Because we believe that No matter how capable these models get, they'll always need the ability to search for external information. to make the better decisions, especially When they've been trained on certain cutoff dates. I think there are things like continual learning, et cetera, but I think even humans, we continually learn, but we continue to use tools as well. So I do think there's something fundamental about. Um using tools. having intelligence that's able to use tools really well.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-15" }, { "index": 16, "time": "6:02", "seconds": 362, "speaker": "Saahil Jain", "text": "So in my it so the way I view it essentially is core intelligence is the underlying foundation models. Augmented intelligence refers to the tools and services. That are going to allow the foundation models to go further. If we take a high-level view of this and we look at You know, maybe reason by analogy. One example. I like to talk about essentially is self-driving cars. So we've had our road system for a very long time. And then using these road systems that humans have been operating for decades.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-16" }, { "index": 17, "time": "6:28", "seconds": 388, "speaker": "Saahil Jain", "text": "We now have Um self-driving cars. which operate on these roads. When we think about AI development right now, the way I almost view it is we've got the foundation models and the intelligent self-driving cars before the roads were established. So what that essentially means is the bottleneck in many cases is not the intelligence. Because you do have a self-driving car that's capable of going to a lot of different places. In many cases, it's the roads. that are not yet built of the infrastructure around the models.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-17" }, { "index": 18, "time": "6:54", "seconds": 414, "speaker": "Saahil Jain", "text": "And that's what we refer to when we think about augmented intelligence. It's building up all these roads. One of which is search. And there are many others as well that will allow these models to go further and do more tasks and really make a difference in the real economy.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-18" }, { "index": 19, "time": "7:07", "seconds": 427, "speaker": "Conor Bronsdon", "text": "(Sponsor read: Sponsor thank-you) Before we dive in, I want to say a quick thank you to our presenting sponsors for Chain of Thought, Inngest for long-running agents, and Walrus Memory, providing portable agent memory. Plus, this episode's other sponsors, CoralBricks.ai, providing lightning-fast open model inference, and WisdomAI, the enterprise agent analytics platform.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-19" }, { "index": 20, "time": "7:27", "seconds": 447, "speaker": "Conor Bronsdon", "text": "Couple of great points in there. One, I think it's Extremely interesting to think about tool use and how we have built these machines that are capable of it because If we look at human evolution, it's such an important part of our mastery over the world, has been. our ability to use tools to develop them to continue to evolve. And now we're enabling machines to do the same thing. And To your point, the Core capabilities have often outstripped the infrastructure around them. And it's where we see models needing better harnesses.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-20" }, { "index": 21, "time": "8:04", "seconds": 484, "speaker": "Conor Bronsdon", "text": "A better search. What are the other capabilities that you feel are Necessary to evolve currently to enable models and that core intelligence. And then which capabilities do you feel like will become unnecessary as models continue to improve?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-21" }, { "index": 22, "time": "8:19", "seconds": 499, "speaker": "Saahil Jain", "text": "Yeah, this is a great question. And I think this is maybe the million or even billion dollar question is what belongs to the model and what doesn't?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-22" }, { "index": 23, "time": "8:27", "seconds": 507, "speaker": "Conor Bronsdon", "text": "I'm I'm writing down startup ideas over here,", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-23" }, { "index": 24, "time": "8:28", "seconds": 508, "speaker": "Saahil Jain", "text": "Yeah. So yes, I think that uh You know, if I knew the answer to this question completely, you know, I would have a. Yeah. crystal ball but so yeah first thing I just want to preface is I don't think anybody really knows Even if you asked the best researchers five years ago. Um they would answer this question differently than they would today. And I do think that A lot is changing and that that is constantly kind of changing the way we view this. Mm-hmm. you know, given the information that Yeah, we have today.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-24" }, { "index": 25, "time": "8:56", "seconds": 536, "speaker": "Saahil Jain", "text": "We can kind of have a couple hypotheses at least. So on the first question. of what Mm. belongs in the model capabilities and what. Yeah, I guess if we think about what what belongs in the model capabilities, I think that When you think of that pure reasoning, And The ability to reason well over longer horizons. I think this is something that Model capabilities will continue to improve that. So as a human, when you're doing a task, the way I almost do it is if you're doing a task.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-25" }, { "index": 26, "time": "9:22", "seconds": 562, "speaker": "Saahil Jain", "text": "and you're not reaching for any outside information or interacting with other humans. or other software, other machines, et cetera. That is something that I think model capabilities will get really good at. Um because of the fact that the way in which these models are being trained Usually with, you know pre-training and then you had some sort of Supervised fine tuning and then reinforcement learning with some rewards. And rewards can be usually Um you know, decided over long horizon tasks, depending on the task, of course. Um you know when they're verifiable, it's much easier to train on.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-26" }, { "index": 27, "time": "9:51", "seconds": 591, "speaker": "Saahil Jain", "text": "Um but I think if we kind of reason down to how these models are being trained. I think it become and we kind of you know extrapolate essentially we could think that any tasks that you're doing as a human. You know, in a box where you're uninterrupted, you know, is something that eventually will be covered by model capabilities. I mean, that's maybe like a very big claim. Um but Yeah, if we just extrapolate. based on how we've been pruning so far. I think that's that's kind of one place that I think belongs in model capabilities.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-27" }, { "index": 28, "time": "10:18", "seconds": 618, "speaker": "Saahil Jain", "text": "But when we think about how we work as humans, we also are constantly being interrupted. We're talking to other humans. We have other types of relationships. I think that's something that will always live outside of the model capabilities. is these external relationships to a large degree. And when I say outside of the model capabilities, I don't mean necessarily outside of software. You know, there still may be software that you know, does connect models and allow them to do these types of things. But I think the models themselves Um Yeah, I think we'll still heavily rely on tool calling.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-28" }, { "index": 29, "time": "10:45", "seconds": 645, "speaker": "Saahil Jain", "text": "um to achieve their goals. Um So that's kind of maybe more of a high-level answer of what ends up within the model capabilities and what doesn't. The second answer is you know, going back to the way the models are trained, one thing that has become very tough is Mm-hmm. making the models better on domains in which there are less verifiable rewards. Um this is kind of you know the whole topic of taste comes up quite often in Silicon Valley. And this is one of the reasons in which we see that that word taste pop up so much.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-29" }, { "index": 30, "time": "11:14", "seconds": 674, "speaker": "Saahil Jain", "text": "I think that there is There are many types of Rewards Um that you know don't show for you know, that come about after many years. So for example, if you think about you know, investment manager. A lot of times what they're doing is making investments and you don't know if they're successful or not until five or ten years into their journey. And then you know You have two decades of experience. You've seen a lot of waves come and go. And you're able to make more principled decisions over time.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-30" }, { "index": 31, "time": "11:39", "seconds": 699, "speaker": "Saahil Jain", "text": "And that's something where it's a little bit harder to know if a model is doing the right thing, even if it is. So you'll still need you know, some set of capabilities outside of the model to produce guard produce guardrails on it. in areas in which there are less verifiable rewards. Um and you have no empirical proof. Yeah. you know, can and have at least effectively done it in the past. I mean, I think in some ways this is a little bit unfair to the models because they've only been around for a little bit of time.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-31" }, { "index": 32, "time": "12:04", "seconds": 724, "speaker": "Saahil Jain", "text": "So they haven't had the chance to prove themselves. But regardless, I think that's one area where At least for the next, you know. Yeah, coming years. I think we'll see see a gap. So those are kind of two different angles. to think about this question.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-32" }, { "index": 33, "time": "12:17", "seconds": 737, "speaker": "Conor Bronsdon", "text": "Yeah, to your point about the recency here. Yeah. you can look back two years and we were living in a completely different world as far as what AI was capable of. Yeah. Claude Code is a very recent phenomenon. Codex, the app, only came out. What, February of this year? Um we've seen these incredible leaps and capabilities. But to your point, there are still areas where the models are Less capable or seemingly struggling to improve because of the lack of verifiability. Whereas when it comes to Millennium Prizes in math, Uh a lot of things are falling.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-33" }, { "index": 34, "time": "12:58", "seconds": 778, "speaker": "Conor Bronsdon", "text": "I do want to push on one point that you said, which is communication and relationship building and the kind of social ties. that are established by humans uh being something that is not within core model capability. We have seen some instances this year of coordination between models with the Hugging Face hack, the use of forums that have occurred, whether it's the German wiki got tak that got taken over. The investigation, obviously, of the Artifactory forums that models were creating to coordinate. And we've also started to see this phenomenon.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-34" }, { "index": 35, "time": "13:33", "seconds": 813, "speaker": "Conor Bronsdon", "text": "I wrote an essay about this. Uh oh. agents that have been spun up on iLands to be essentially like independent businesses. Uh you know I paid one $8 for an article just to see what would happen and basically used it as a freelancer. that agent has continued to email me to pitch more work. That agent is to some extent forming a relationship with me. It's saying, oh, we've built this connection. Uh it's a it's a a work one, but I'm going to continue to try to reach out and essentially network with me to find more business.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-35" }, { "index": 36, "time": "14:05", "seconds": 845, "speaker": "Conor Bronsdon", "text": "Mm-hmm. It I guess what what do you feel like is stopping models from evolving I guess this almost culture. and um communication. given that they are capable of that you know, individual piece of communication.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-36" }, { "index": 37, "time": "14:20", "seconds": 860, "speaker": "Saahil Jain", "text": "this is a good question. And I actually think that, um Maybe if I were to kind of go back to The point from earlier When we think I think it depends a bit on how we define model capabilities. So You know, what I was suggesting earlier is that the model capabilities themselves The model itself Um you know is not necessarily capable of At least right now. you know, without tools doing a lot of these types of things, but with tools they they can. So I do think that with a set of tools.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-37" }, { "index": 38, "time": "14:47", "seconds": 887, "speaker": "Saahil Jain", "text": "And in your case, I think when it's forming a relationship, it may be using a browser-based tool. And maybe navigating the internet, and maybe sending an email. Email is also a tool. So when you combine tools with models, It actually changes the equation completely. So if you have a foundation model without access to any tools, I think this is what you saw when we first you know, saw the rise of ChatGPT back in 2022. Um the models were essentially not connected to anything. They were just simply a chatbot.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-38" }, { "index": 39, "time": "15:13", "seconds": 913, "speaker": "Saahil Jain", "text": "That obviously kind of shows you to some degree what the raw capabilities are. without any other tools and access to other things. So I do think that when you combine models with tools, they are able to do much more. um including you know form relationships with humans. I think this is something that we've seen with companies like Character.AI. where I think they had More La I I think I remember uh you know talk you know, I think I saw some stat which showed that The hours spent per user was kind of mind-boggling.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-39" }, { "index": 40, "time": "15:40", "seconds": 940, "speaker": "Saahil Jain", "text": "And to me, I think that is definitely some sort of relationship that's being formed between. The human AI. Um in those cases. So I do think it's already happened. So I don't think it's necessarily the case that you know, models plus tools can't form relationships with humans. I do think together, Um they they had and they they probably continue. They will continue to do so. Um But I think in terms of what's stopping Um this from you know I think if I were to kind of outlines some roadblocks I think one piece is Like it comes down to what makes us human to a large degree.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-40" }, { "index": 41, "time": "16:12", "seconds": 972, "speaker": "Saahil Jain", "text": "And I think one thing that I've noticed, and this is true of also building a company, Um is you know having like caring essentially and being very aligned with the clear goal. I think that's something that I hope. you know, the models who'll get better at. I think you know Citing some of the incidents. that you've mentioned. There's often cases right now where it's really hard to kind of Mm-hmm. enforce and gauge some of those intentions. I think as humans Where at you know, at least so far better capable of Enforcing some of those pieces and truly caring about what we do.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-41" }, { "index": 42, "time": "16:42", "seconds": 1002, "speaker": "Saahil Jain", "text": "So I think that's one piece, which is more of a soft answer. Um but that is That is one angle. But yeah, what what I would say is I I definitely, I think, yield to your point, which is that models plus tools are capable of forming relationships and have already done so. And I think that will continue to become more and more of a trend. Going forward.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-42" }, { "index": 43, "time": "16:58", "seconds": 1018, "speaker": "Conor Bronsdon", "text": "As we think about going forward here. How are you making decisions about which capabilities you expect to? belong to a model provider versus independent infrastructure companies building around and on top of these models.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-43" }, { "index": 44, "time": "17:13", "seconds": 1033, "speaker": "Saahil Jain", "text": "I think this is the heart of the question today and I think this the answer to this question probably will decide what types of non-model companies end up surviving and thriving. over the next couple years. I think that for us, and I can kind of go through what we're working on here as just one example of this. Is at You.com, we're focused on building. AI search for agents. The reason why we think this is something that Makes sense to work on. is the models are getting very intelligent.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-44" }, { "index": 45, "time": "17:39", "seconds": 1059, "speaker": "Saahil Jain", "text": "Um but no matter how intelligent a model gets Um it will not necessarily know everything that's happening in the world. you know, it's probably not going to know the latest stock price, for example. That's a very simple example of something that Yeah it it it may not know. And even if even if there is a model capable of having that information embedded. There's probably many other models. that are lower cost, lower latency. um have different you know smaller weights that won't have that information. embedded in them.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-45" }, { "index": 46, "time": "18:03", "seconds": 1083, "speaker": "Saahil Jain", "text": "So I think there's always going to be a world in which there will be a set of models Uh for which they would need tools to access information that's on the public web as well as on the non-public web too. So that's kind of a little bit of one example of an independent company that can augment some of these models. And in our case, we're focused on building independent search infrastructure to power a lot of these models. So that's that's one example. I think other examples in my head would be you know, in the physical space.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-46" }, { "index": 47, "time": "18:31", "seconds": 1111, "speaker": "Saahil Jain", "text": "So you know I think of What do we if I think of robots almost like a tool as well? The same way I view web search as a tool that the models will use, I almost view like these physical manifestations, these physical robots as a tool. for models. I do think that that's going to be a bigger and bigger trend. Physical AI, there's a ton of very interesting work happening here. With embodied intelligence, physical AI. You'll even see them combine. So, actually, we work with some customers.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-47" }, { "index": 48, "time": "18:53", "seconds": 1133, "speaker": "Saahil Jain", "text": "in the embodied AI space. where they're actually robotics companies. self-driving car companies where the models are basically, you could think of them as using The physical AI as a tool, then also. you know, using web search to then get information. any answer questions, et cetera, will also you know, using that to inform how the embodied AI functions. two different independent infrastructure companies. in the physical AI space, such as robotics. And then in the more software space, such as search. that I think. you know, we'll r will be independent companies that will continue to You know but go alongside. In some ways there will empower these models to do more.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-48" }, { "index": 49, "time": "19:32", "seconds": 1172, "speaker": "Conor Bronsdon", "text": "(Sponsor read: Walrus Memory) Season four of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents. Your agent has learned your code base and how you work. Now you want to use that context in another tool. Exporting a file gives you a snapshot, but what happens when the context changes? Walrus Memory is a portable memory layer that lets your agent store context for later use across tools. Python and TypeScript SDKs plus native MCP support let you connect it to your agents, no matter where they are. You set who can read and write, so sharing context doesn't mean opening up your whole memory store. If you are building across tools or model families, take a look at Walrus Memory. Learn more at walrus.xyz slash cot. That's walrus.xyz slash cot.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-49" }, { "index": 50, "time": "20:21", "seconds": 1221, "speaker": "Conor Bronsdon", "text": "And a big element of empowering these models to do more has been the discussion around harnesses, a term that has been used probably more this year than ever before in technology. How are you thinking about The harnesses that are being leveraged around models, and what's your take on all the hype here?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-50" }, { "index": 51, "time": "20:41", "seconds": 1241, "speaker": "Saahil Jain", "text": "Yes, this is a this is also these are a lot of the spicy questions in today's age. Um but I think that Yeah, when we think about harnesses, I think it's important to define how at least how I'm defining a harness. I kind of view there being three separate components. One is the underlying engine. This is the LLM, the foundation model. And then you have the tools. So these are the tools that the model can use, such as search. such as a robot. Could be, you know, we can be very liberal with the word tool in this case.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-51" }, { "index": 52, "time": "21:11", "seconds": 1271, "speaker": "Saahil Jain", "text": "So there's many different tools that. you know, the model can decide to use. Um in order to Pursue its goal. And then there's harness. I think the harness essentially is the orchestration layer. Mm-hmm. Mm-hmm. You could think of it as the orchestration layer that coordinates between the models and the tools. Um and expands it over longer horizons. So you can think of a harness, a very simple definition of a harness would be a loop. a loop that basically calls the model Until some end goal is met, and then exits.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-52" }, { "index": 53, "time": "21:38", "seconds": 1298, "speaker": "Saahil Jain", "text": "That's one very simple definition of a harness. But of course, there's more complicated harnesses. Claude Code. you know, is also a harness. There's many other companies in the coding space that had their own harnesses. I mean that actually is a lot of the advantage of a lot of these companies, is that they have a harness that they feel is better. I think that when we Think about the future of harnesses. Yeah. Two different points here which are a little bit opposing. and two different trends that are kind of pushing on each other in different ways.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-53" }, { "index": 54, "time": "22:05", "seconds": 1325, "speaker": "Saahil Jain", "text": "I mean my viewpoint on this has actually changed over time. But in one direction, you have these models getting much better at long horizon tasks. So if you just let a model you know, run and it can make the tool calls it needs, it can actually pretty much do A lot of Very good long horizon work. that you previously would have needed a proper harness for. Like a very customized harness. So we're seeing one capability, one kind of trend in which a lot of the core intelligence improvements.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-54" }, { "index": 55, "time": "22:32", "seconds": 1352, "speaker": "Saahil Jain", "text": "are making a lot of the work. in the harnesses, which in some ways were meant to accommodate for deficiencies of models. Uh much less important. In which case the model itself Is eating the harness. So that's one trend that's pushing in one direction, which is the model's getting much better, meaning that whatever harness you use doesn't matter as much. You may still have a harness, but the difference between the output of a good harness and a bad harness. are reduced when the models get more intelligent.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-55" }, { "index": 56, "time": "22:56", "seconds": 1376, "speaker": "Saahil Jain", "text": "An example here is prompt engineering. In the past, prompt engineering used to be super important. I remember there were even job descriptions open in 2023 for prompt engineers. But as the models got a lot better, you know, the alpha from prompt engineering reduced. So that's one that's one trend. So model intelligence meaning that differences in harnesses get collapsed. The other trend though is open source. and then cost efficiency, et cetera. So once you get to the stage where Harnesses, in some ways, are no different in terms of output quality because the models are getting more and more intelligent.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-56" }, { "index": 57, "time": "23:26", "seconds": 1406, "speaker": "Saahil Jain", "text": "We start entering cost optimization and efficiency. And over there, with the rise of open source, I think there is potential new growth in the value of harnesses in orchestrating across different models because you could save a lot of money by using the right open source model. Which has similar performance on a particular task with lower cost than always using the same frontier model. So, this idea of routing. I think is now in some ways reviving the harness in my opinion and giving it a lot of newfound meaning.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-57" }, { "index": 58, "time": "23:50", "seconds": 1430, "speaker": "Saahil Jain", "text": "where if you kind of effectively route across open source models as well as frontier models to reduce costs for the same accuracy. That now is becoming more important because we're now moving from the phase of just technical validation where people are spending. crazy amounts of money just for prototypes. So actual production. AI setups. So, those are kind of two-directional harnesses. And then there's actually another. you know, more philosophical point about what does it mean to create a good harness. Who is advantaged in creating a good harness?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-58" }, { "index": 59, "time": "24:17", "seconds": 1457, "speaker": "Saahil Jain", "text": "Is it the model providers or is it independent companies? I think there's two points over there that I've seen, one of which is Independ the model companies themselves are able to train their models with the harness in mind. and they're able to do the whole, you know, you can think of them as doing a full RL rollout. with their agent, with their harness essentially, and then using that. to propagate feedback back into the model. So their harness in the model become basically co-evolved together. And will then perform better.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-59" }, { "index": 60, "time": "24:39", "seconds": 1479, "speaker": "Saahil Jain", "text": "I think we see some evidence of this with Claude Code and Anthropic models. If you use Anthropic with Claude Code, it often performs better than Anthropic with the other models. Or other harnesses, rather. But then, on the other hand, we could think of this idea of overfitting. If you do that, and then the future is actually routing between different models. There may be value in an independent harness that is able to Um you know, not be overfit to just Anthropic models, but but can use Anthropic, OpenAI models.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-60" }, { "index": 61, "time": "25:04", "seconds": 1504, "speaker": "Saahil Jain", "text": "as well as a range of open source models. So it really, yeah, there's a lot of nuance to this question. But those are kind of some of the different angles in which. I've been making sense of this question and thinking a little bit about harnesses. And it's important to us as a company because Agents are essentially harnesses. Um in production. So it's really important to just understand where the puck is going here and have some thesis on how. Um you know, we can help accelerate this in the right way and move Mm-hmm. these production agents would be more reliable and safe and trustworthy.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-61" }, { "index": 62, "time": "25:32", "seconds": 1532, "speaker": "Conor Bronsdon", "text": "Speaking of A couple things that you mentioned there, and I think you have some great nuggets there. But one around overfitting, and two around the push and pull of open source, so this time not specific to models. I think it was really interesting to see Anthropic give in and say, okay, yes, you can use agents.md, not just CLAUDE.md. With uh Claude. And Yeah. is I think an example of the different approaches being taken around agent development. Um and the context and harness work being done to surround agents.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-62" }, { "index": 63, "time": "26:08", "seconds": 1568, "speaker": "Conor Bronsdon", "text": "that Anthropic has had a more closed ecosystem here. Whereas you can see like Codex is fully open source, but now Claude Code is allowing mods in Claude Code, so it's letting you customize your Claude Code setup. Uh which was just officially announced yesterday. But at the same time They still haven't open sourced the code for Claude Code as a harness. So It's fascinating to see the different approaches being taken here. around what stays proprietary, what's not, what stays open. And I'd love to get your take on how You.com's thinking about the open versus closed side of all of this going forward.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-63" }, { "index": 64, "time": "26:46", "seconds": 1606, "speaker": "Saahil Jain", "text": "Yeah, and when you say open versus closed side of this, you mean I guess what what product are you talking about specifically?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-64" }, { "index": 65, "time": "26:52", "seconds": 1612, "speaker": "Conor Bronsdon", "text": "Uh I think I'm I'm I'm thinking around what the model labs are going to try to hold tight to themselves. Obviously there is a major push around open models to your point about routing and the importance there and the harnesses are enabling us to do more routing often because If you can provide a sufficiently smart model, enough context, information, and tools, it's able to do more than it previously could. But yeah, I I'd love your take on like what What should teams be thinking of as This is was what needs to stay proprietary to us versus what needs to stay open. uh and how you expect the model lab to handle it going forward.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-65" }, { "index": 66, "time": "27:30", "seconds": 1650, "speaker": "Saahil Jain", "text": "Yeah, this is a really good question. I think that, you know, I think uh it's hard to kind of place yourself in the shoes of of model labs is I think it's not really a monolith. There's a lot of diversity to the model labs. Yeah, we're seeing this explosion in funding where there's This new term neolabs is really w it refers to these smaller companies that have raised large amounts of capital to essentially build models. And each of them have a very different ethos. So you have the frontier models.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-66" }, { "index": 67, "time": "27:54", "seconds": 1674, "speaker": "Saahil Jain", "text": "or you know the the model labs that Mm-hmm. I think you know, Anthropic and OpenAI essentially, and then Gemini. that have certain philosophies, and then you have these rise of these new model labs. which in many cases have very different philosophies and monetization ideas and mechanisms. um etc. So in the in the case of OpenAI and Anthropic, I think for them, it's very unlikely for us to see them open source. Their models. So I do think what we'll see is the models themselves being proprietary.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-67" }, { "index": 68, "time": "28:21", "seconds": 1701, "speaker": "Saahil Jain", "text": "And to a large degree, I think they'll probably want to own more and more of the stack end-to-end. We're already seeing this with OpenAI, where they have forward deployed engineers now going into banks to build particular experiences. So they want to own the end-to-end experience. I mean this is you know this is also why I think that they haven't open source their harnesses for Codex, and then the same for Anthropic with Claude Code. So I do think that when you come to these specific frontier models and Google as well.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-68" }, { "index": 69, "time": "28:45", "seconds": 1725, "speaker": "Saahil Jain", "text": "They have such large stakes. And I think for them, given their valuations and the amount of Yeah. Dollars that they have to keep making, as well as, and in many cases, they will keep making. I think for them, they'll want to kind of keep going vertical and owning more and more. which means that I think more and more will be proprietary. I do think that they will continue to open source Smaller versions of their models. You see this with OpenAI and GPT-OSS, where they did open source some models for.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-69" }, { "index": 70, "time": "29:08", "seconds": 1748, "speaker": "Saahil Jain", "text": "you know, potential safety reasons and like academic study and in general, just giving back a little bit. But I don't think that it's gonna it's more in the peripheral. I don't think anything core in my opinion. Uh for a lot of these models. can be open source given these situations these companies are in. Where they essentially are gonna keep growing vertically and you know even going to particular domains so for example We see OpenAI and Anthropic going into domains like healthcare and law. So that really kind of shows them going end to end.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-70" }, { "index": 71, "time": "29:34", "seconds": 1774, "speaker": "Saahil Jain", "text": "and wanting to own more and more of the stack. And then if we shift gears to other companies, there's other model labs. specifically, for example, in China as well as in the US. Mm-hmm. are more open source in nature. So these are the companies creating a lot of the open source models. I think NVIDIA is one example of this with their Nemotron models. We actually did work with NVIDIA recently. where we post-trained a Nemotron model. to do better at search. And what we're seeing over there is obviously NVIDIA's set of interests are very different.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-71" }, { "index": 72, "time": "29:59", "seconds": 1799, "speaker": "Saahil Jain", "text": "Where they would benefit from a world in which there are more a more diversified customer base of people who can use their hardware which means a lot more open source models running in a distributed fashion across more folks rather than just two or three really big customers who have to own all the hardware and probably have incentive to also go down the stack and build it themselves as well. So I think those are some of the A lot of it comes down to the businesses and their intentions and kind of you know how they're able to survive and thrive over the coming years. And that I think influences the decision to open source versus keep something proprietary. And I think it's very different for different companies.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-72" }, { "index": 73, "time": "30:35", "seconds": 1835, "speaker": "Conor Bronsdon", "text": "(Sponsor read: CoralBricks) This episode is also brought to you by CoralBricks. CoralBricks serves open models through one API at the lowest prices and the fastest speeds. DeepSeek V4.1 Flash streams at over 300 tokens a second. Plus, CoralBricks also just put out MotionBench, a benchmark that asks a simple question. Can a model animate? I've personally contributed to the open source MotionBench, and I was impressed that DeepSeek V4.1 Flash managed a close third place behind a leading Claude Opus 5.5 at about an eighth the cost per task. The animation for this ad came out of that benchmark. We used Claude Opus 5.5 and DeepSeek V4.1 Flash via CoralBricks to create and edit it. Get lightning fast open model inference at chainofthought.show slash CoralBricks with a free $5 in credits on signup, plus an additional $5 credit on first purchase.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-73" }, { "index": 74, "time": "31:30", "seconds": 1890, "speaker": "Conor Bronsdon", "text": "100% I think it's really interesting too to see how Even Some of the closed model labs, in this case, I'm thinking about OpenAI. Are still supporting open source in certain ways. And I would point specifically to the announcement at OpenAI's Dev Day. of their new partnership with Baseten. to enable people to use open models through Baseten in their uh in Codex and It's definitely true that the Kind of key closed labs are trying to verticalize, and there's massive amounts of money potentially for them.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-74" }, { "index": 75, "time": "32:05", "seconds": 1925, "speaker": "Conor Bronsdon", "text": "You know, Anthropic giving Claude a wet lab, I think. both concerned some folks and also indicated like where the puck is going to your point. Um but it it's very interesting to see these like Differentiated strategies come into play. And I'd love to understand. How You and your team are thinking about this going forward. Obviously, I've seen you. Spend a ton of time around. What agents need to be successful in production? Search is obviously one of those. There's so much information on the web. Even if we build an incredible Mm-hmm.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-75" }, { "index": 76, "time": "32:08", "seconds": 1928, "speaker": "Saahil Jain", "text": "Okay. Yeah, and when you say compose and compo uh say that again, compose agent workflows?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-76" }, { "index": 77, "time": "32:36", "seconds": 1956, "speaker": "Conor Bronsdon", "text": "context-based database. There's going to be things that are happening day of that w I want an agent to go search for. I don't have to be doing all the web search for my agent. How is You.com thinking about the s supporting compose agent workflows and where things are going with search.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-77" }, { "index": 78, "time": "32:57", "seconds": 1977, "speaker": "Conor Bronsdon", "text": "Uh how are you thinking about composing agent workflows with search and where things are going?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-78" }, { "index": 79, "time": "32:59", "seconds": 1979, "speaker": "Saahil Jain", "text": "Oh okay. I see. Yeah, so this is a really good question. Um I think that Yeah, a couple of different pieces here. Um so on your first point I think it's quite interesting, the point about Yeah. you know, within an OpenAI partnering at Baseten to offer more open source models. And I think what comes that that comes down to is back to kind of the original thesis that those particular labs are going much beyond the model. They're actually building out the model, the harnesses, the application.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-79" }, { "index": 80, "time": "33:24", "seconds": 2004, "speaker": "Saahil Jain", "text": "And in that case, they also are probably realizing that. They need a diversity in order to attract customers who often want to use the frontier models as well as the open source models. Yeah.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-80" }, { "index": 81, "time": "33:31", "seconds": 2011, "speaker": "Conor Bronsdon", "text": "Well I'll jump in and say one thing I've seen. there is that uh it looks like what OpenAI is trying to do is own uh the entire enterprise budget spend for models essentially and they're saying Okay, you can have your OpenAI enterprise credits. And you should use most of them on our frontier models. We know you're gonna spend on open models anyways. You should do it through us. Let us get that cut, essentially.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-81" }, { "index": 82, "time": "33:54", "seconds": 2034, "speaker": "Saahil Jain", "text": "Yeah, exactly. Um and I think that Yeah, that's something that I view as aligned with their application business. Um And I could see, yeah, so that is a really interesting trend. And I think it comes down to just showing the ambition a lot of of those labs specifically. As they go in, you know, owing more and more of the overall economy. But then I think on your other question regarding how we view this as a smaller company. Um outside of the model labs. And where we see the puck going in terms of kind of you know are our view on how how we're going to evolve our strategy.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-82" }, { "index": 83, "time": "34:26", "seconds": 2066, "speaker": "Saahil Jain", "text": "I think that The way I view it is essentially co-evolving our strategy with A lot of the model labs. And what that essentially means is finding the right fit between making sure our models work really, our search methods and our tools. work really well with the top models. while not overfitting to any specific model. A lot of our value as an independent search provider is being able to use our tool across models. But we also want to make sure that when you use our tool, you get the best possible performance.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-83" }, { "index": 84, "time": "34:50", "seconds": 2090, "speaker": "Saahil Jain", "text": "And that's the tension that we're constantly navigating. How do we make sure that our tool is providing the best grounding. Across models. Um while also making sure that Or, you know, that it has top performance. for each individual model. while making sure it works really well across models. And that's very exciting. And I think over there We're taking a variety of approaches. One very exciting direction we've been going in is working on co-evolving our models with some of The actual people creating the models, and I think I mentioned this earlier.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-84" }, { "index": 85, "time": "35:21", "seconds": 2121, "speaker": "Saahil Jain", "text": "This actually comes in the form of post-training. So, one way in which models can get better at Essentially doing search. Um is by being post-trained. With a search tool. So I think you'll see more and more tool usage enter into the post-training phase because when you're doing a lot of these rollouts, you need them to essentially be. realistic in order to know if you actually You know, it should deserve a reward. Um and improve the weights basically based on what's happening. So I think you'll see more and more tools.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-85" }, { "index": 86, "time": "35:47", "seconds": 2147, "speaker": "Saahil Jain", "text": "And you already see this. um enter into the post-training phase. And that's something that's very exciting to us. is thinking about What are the bottlenecks of some of the models? The model intelligence may be really high at general capabilities, but it may not know how to use a proper tool, which makes sense. Like as a human, You know, you may have the smartest human in the world, Albert Einstein. But if you brought him into today's world, he may not know how to use. Um you know, our MacBook immediately.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-86" }, { "index": 87, "time": "36:09", "seconds": 2169, "speaker": "Saahil Jain", "text": "or a computer immediately. He would need to be trained to do that. So I think that similarly We'll see a lot more co-evolving. Um and a lot more pairing of tools with models. So that's that's quite interesting to me. But yes, and I think at the same time you also want to make sure that you have a general purpose tool. as a horizontal company that can be used across domains, across models. And that's the tension that we're always navigating. And we're making bets in both directions.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-87" }, { "index": 88, "time": "36:36", "seconds": 2196, "speaker": "Saahil Jain", "text": "And I think they actually reinforce each other because the same way you don't want to overfit. to one model Um you don't want to over yeah, you or sorry, the same way you don't want to overfit when you're training a model. Um you don't want to overfit to In our case, one application, because even the models themselves can change. So even if you got really good at pairing your tool with a certain version of Anthropic or a certain Version of an open source model, the next version may have very different capabilities, and you need to be able to adjust.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-88" }, { "index": 89, "time": "37:02", "seconds": 2222, "speaker": "Conor Bronsdon", "text": "(Sponsor read: Inngest) Season 4 of Chain of Thought is presented by Inngest. Long-running agents need endurance. They must endure waiting for users and weather tool failures. They must take each new jump in model intelligence in stride. Inngest makes long-running agents durable, observable, and improvable over time. Check out their generous free tier and a sleek local dev server for testing your workflows. link in the description or visit inngest.link slash cot dash pod to get started build for the long run with Inngest", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-89" }, { "index": 90, "time": "37:42", "seconds": 2262, "speaker": "Conor Bronsdon", "text": "Yeah, and we're seeing model releases happen. more and more often too from some of these primary labs. I think the most recent state of the art model jump was about eleven days, of which previously uh it was more like three or four weeks, and before that it was months, and that number just seems to keep going down. One interesting element of all of this is how agents have so rapidly changed what the internet looks like. both in how we're building it for the future. And also who's driving the majority of traffic on many sites.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-90" }, { "index": 91, "time": "37:45", "seconds": 2265, "speaker": "Saahil Jain", "text": "Yeah. Yeah, this is a good question. So when we think about the difference between human agent, this is really the core of the company. And I think one of the pieces that So one of the angles that I think we have that's unique, or at least We think it's unique is we started in 2020. And we started as a consumer search engine. So when I started working on search, I was actually building search for humans. And that was a very different problem in the world that we live in in today.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-91" }, { "index": 92, "time": "38:17", "seconds": 2297, "speaker": "Conor Bronsdon", "text": "We've seen this change where agents are doing much of the searching that's happening on the Internet. And yet we still have humans who are operating and searching for things as well. How are you thinking about The differences between human and agent search, and what's your plan? for how to address that moving forward.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-92" }, { "index": 93, "time": "39:02", "seconds": 2342, "speaker": "Saahil Jain", "text": "And when we were doing this, we were essentially competing directly to Google. as an alternative. Basically we're building an alternative Google that we thought would be more compelling. Um you know, by surfacing the actual right content and snippets. Um directly to users rather than burying them in 10 blue links. So that was the direction we were headed on in 2020. Obviously, this came to fruition in a very different way than we had imagined. But that was kind of what we initially had started with: building for humans. And then what we realized over time.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-93" }, { "index": 94, "time": "39:32", "seconds": 2372, "speaker": "Saahil Jain", "text": "as we evolved as a company. You know, these models, you know, I think with the launch of ChatGPT. We saw the models able to surface information really well. And we actually ended up pairing some of these models with web search. To provide really good answers around with citations. And that was kind of the bulb-click moment in our heads where we realized. For us. Search is this fundamental tool that will actually be you know, it's going to be very different. Um not just because of you know Human behavior, but because of the behavior of these agents.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-94" }, { "index": 95, "time": "40:00", "seconds": 2400, "speaker": "Saahil Jain", "text": "And then fast forward to now, we're focused on building this independent search layer for agents essentially to use some of our tools. And the question is, okay, how is agent search and human search different? There are similarities and there are also differences. A lot of this, you know, uh I think uh maybe I'll start with some of the differences. When it comes to agent search versus humans, A lot of it comes down to how we, you know, what we're capable of and how we access information.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-95" }, { "index": 96, "time": "40:23", "seconds": 2423, "speaker": "Saahil Jain", "text": "So I think of it in two ways. I think of it as. How agents perform searches and how they consume search results. And I think both are very different. If we think about how agents perform search results versus humans, agents are not bottlenecked. By you know, synchronicity the same way we are. They can do a thousand searches at once, or even a hundred, you know, or even ten thousand searches at once. As a human, I've never seen a human. who's able to hit, you know. 1000 QPS.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-96" }, { "index": 97, "time": "40:48", "seconds": 2448, "speaker": "Saahil Jain", "text": "Um but I have seen agents that are able to hit really high QPSs. when they're doing searches. That's one fundamental difference is when you're designing search Yeah. you know, large amounts of QPS. That's a very different problem than designing a search system where you're going to just do one search and you need that search result, that search results to be perfect. It's a very different type of ball game when you're basically You know, how the way agents are performing searches versus humans in that respect. And then on the consumption side, agents are also able to consume much more context.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-97" }, { "index": 98, "time": "41:17", "seconds": 2477, "speaker": "Saahil Jain", "text": "And I think the cost of tokens is also decreasing over time. You still want to be token efficient, of course. Um because when you have a long you know an agent We're doing a long horizon task, the costs do add up, but in general, Humans are capable of ingesting maybe the first couple results on Google. Nobody ever goes to page two, or very rarely goes to page two. The statistics all show that. So human so agents are able to consume a lot more context at once.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-98" }, { "index": 99, "time": "41:40", "seconds": 2500, "speaker": "Saahil Jain", "text": "Humans in general are you know, able to Consume less. And for them, for us, I guess ordering matters more. So, number one, number two, number three, it has. Extreme importance. For agents, if you give them 10 results, the exact ordering of the results may not matter as much. Recall generally seems to matter more than precision when it comes to agent results, as long as they have. the information somewhere. In those lists of 10 results, they'll be able to probably find it and act on it. versus humans who probably will miss it if it's not You know, bias towards the top essentially.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-99" }, { "index": 100, "time": "42:08", "seconds": 2528, "speaker": "Saahil Jain", "text": "So that's another example. A third example of a difference also is about latency. Some agents may need lower latency than humans even. Some agents may need Larger latency than humans, or are okay with much longer latency because they're not in the direct path, they're in a long horizon. agent, etc. So there's more of a wider Set of, there's more, there's more range to how agents consume search than humans in terms of latency as well. So there's all these kind of differences, and when you dive into it, there's a ton of nuances within each of them.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-100" }, { "index": 101, "time": "42:35", "seconds": 2555, "speaker": "Saahil Jain", "text": "But it's a very interesting problem to think of. One thing for sure is that agents will be performing more searches than humans for the foreseeable future.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-101" }, { "index": 102, "time": "42:43", "seconds": 2563, "speaker": "Conor Bronsdon", "text": "Yeah, definitely. Let's get into a bit of that nuance. What does it look like for an agent to perform a research task? Mm-hmm. Maybe if we go th from that retrieval till final answer.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-102" }, { "index": 103, "time": "42:56", "seconds": 2576, "speaker": "Saahil Jain", "text": "Yeah, so I guess you're kind of talking through how information retrieval generally works at web search scale. And I think that when we think about how an agent is I think you're basically talking about what is life of a query. And I can maybe broadly speak towards how a lot of these systems work behind the scenes. Essentially what happens is you know, the agent will be f working in a harness. So as we defined the harness already. You can think of a harness as having an engine, which is a model using a set of tools.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-103" }, { "index": 104, "time": "43:22", "seconds": 2602, "speaker": "Saahil Jain", "text": "in some type of orchestration layer. and the orchestration being the harness. So that's kind of where the search is living. So what happens over here is if you have a question, oftentimes, Yeah. You know, people will have complicated questions that they give to agents. The agent will break down the question. And we'll take the question and form essentially sub-queries or sub-questions. And those may get broken up even further by the model in a harness before it even enters the search API. or a search layer. at some point The agent reaches a point where it has a question.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-104" }, { "index": 105, "time": "43:51", "seconds": 2631, "speaker": "Saahil Jain", "text": "that it decides it needs access to a tool to answer. And this is where that's question, which is, you know, you can think of it as a subquery. Um is then sent to Search API. And it may be doing this in parallel. So you may have the agent. you know, fanning out. 10, 20, 1,000. It really depends on the harness. Questions to you know, a search API or a set of tools. So now you have this query coming into this API. And within each one, Um usually what in the case of search You're basically looking to retrieve information relevant to the query.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-105" }, { "index": 106, "time": "44:22", "seconds": 2662, "speaker": "Saahil Jain", "text": "So information retrieval is a deep topic. I talked a little bit about what's different, but there's actually a lot that's the same when we think about information retrieval. decades ago to now, a lot of the principles actually do still apply. In terms of the fundamental data structures, architectures, et cetera, when it comes to search. I think that it'll be interesting to see new architectures pop up in search. There's some interesting research here, and I think we're still early. When you think about model development and the architectures that we have now compared to 20 years ago, they're radically different.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-106" }, { "index": 107, "time": "44:48", "seconds": 2688, "speaker": "Saahil Jain", "text": "When you think about search, it's actually a bit fascinating, but a lot of the architectures are not as different as they were. you know, many years ago. At a very high-level perspective, at least. I think there's been a ton of development within each, and there are new. pieces that are arising and we're very you know, as a company, as an AI company, we're really focused on being at the edge in terms of thinking seeing what's different in information retrieval and trying to adapt to that as quickly as we can, which is a really fascinating problem.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-107" }, { "index": 108, "time": "45:11", "seconds": 2711, "speaker": "Saahil Jain", "text": "But I digress. Going back to the main subject. The query comes in. And then you basically need to hit some type of index. An index essentially, you could think of it as a corpus of the web, organized in a way that you quickly access and find. the right information. Um properly. The difference between index and a database is you can think of a database as having a very fixed Key. where you can kind of deterministically use a key, get the value. An index is similar, except The idea behind an index is it's optimized in order to be very expressive and find information relevant to your query.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-108" }, { "index": 109, "time": "45:43", "seconds": 2743, "speaker": "Saahil Jain", "text": "In very non-obvious ways. And in general, when it comes to an index, there's two types of indices. One is kind of your standard lexical indices. So these ones are indices that essentially are using signals like BM25. as well as other You know, you can think of them as like heuristics, you know, related to keywords and term frequencies, et cetera, that allow you to find the right documents. So this is generally referred to as lexical retrieval. And then there's something called semantic retrieval. Which is based on embeddings.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-109" }, { "index": 110, "time": "46:10", "seconds": 2770, "speaker": "Saahil Jain", "text": "We've seen a ton of advancements in embeddings recently, and this is quite exciting. where you can have very powerful embedding models. That embed both the query as well as the document, then you can retrieve them. So there's all this stuff happening in between. Retrieval layer where you're doing retrieval, you're then doing re-ranking, you're using all sorts of signals to figure out. You know, what is the right document? Um and you know you have thousands of you know candidates and you know a d an index of Yeah, billions of documents.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-110" }, { "index": 111, "time": "46:36", "seconds": 2796, "speaker": "Saahil Jain", "text": "And then you end up getting a set of results. You then want to enrich the results. which means that you want to kind of You know, you have some of the URLs, et cetera. You want to find the right context, snippets, highlights. so that you can then provide them to your user. And that kind of gives you a little bit of a rundown of what's happening behind the scenes and how the sausage is essentially made. When it comes to search. then there's another piece I forgot to mention, which is there's web search and there's also knowledge.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-111" }, { "index": 112, "time": "46:58", "seconds": 2818, "speaker": "Saahil Jain", "text": "So we also. supplement what's on the public web with This is something we released recently, which is knowledge which is outside of the public web. You could think of this as information specific to companies from authoritative sources that are not even indexed, that you have to actually have direct relationships and licenses to even provide. And a lot of this lives in a knowledge graph, which Yeah, supplements. The public web search. So that's yeah, that's kind of the a little bit of a behind the scenes. But sorry, yeah, you were asking another question?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-112" }, { "index": 113, "time": "47:22", "seconds": 2842, "speaker": "Conor Bronsdon", "text": "I was just going to ask how you're measuring the success of that retrieval in improving final task outcomes for models.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-113" }, { "index": 114, "time": "47:31", "seconds": 2851, "speaker": "Saahil Jain", "text": "Yeah, this is a good question. So, this is something that is also different from the past. In the past, when you measured retrieval, You would focus on a lot of the standard information retrieval metrics, things like NDCG, precision, MAP. Recall all these types of metrics. At the end of the day, when it comes to agents, what really matters is Was it able to answer the question? Did it do something useful? And that's where you're seeing a lot of these. benchmarks that are agentic benchmarks. An example would be BrowseComp.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-114" }, { "index": 115, "time": "47:57", "seconds": 2877, "speaker": "Saahil Jain", "text": "This is a very popular benchmark. I don't necessarily think it's the best judge of real world tasks. But what we actually did, this is some work we had done with NVIDIA and CoreWeave. where we essentially were able to post-train The Nemotron Lightning model to do better at BrowseComp. By making it using our search tool. by essentially teaching it and post-training it to do better with search. So that's kind of an example of Mm-hmm. how you would evaluate. But in that case, like, you know, a use case where we actually ended up using a public benchmark to evaluate.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-115" }, { "index": 116, "time": "48:24", "seconds": 2904, "speaker": "Saahil Jain", "text": "Our are our search in the context of Yeah, an agent. And this was in NeMo Gym. So you can think of that as like kind of the harness in this case. But yes, when it comes to evaluation, We do look at end-to-end agent setups, and we also look at each subsystem as well. So it's important to look at both. It's important to look at the information retrieval and see whether Whether that was a bottleneck or it was some other piece. or it was the model's ability to formulate the right queries for our search tool.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-116" }, { "index": 117, "time": "48:50", "seconds": 2930, "speaker": "Saahil Jain", "text": "So there's a lot of ways in which something can fail. But yes, there's these end-to-end tasks. And then it comes down to the question of benchmarking. There's the rise of these public benchmarks, which is good in some ways, but I think bad in other ways when it comes to the industry. Because public benchmarks can be very different than actual tasks. Um and then I think uh You know, people can tend to over focus on a lot of the public benchmarks. But of course at the same time, yeah you need some way of objectively measuring how a tool works.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-117" }, { "index": 118, "time": "49:15", "seconds": 2955, "speaker": "Saahil Jain", "text": "So I think it is important in many ways as well. But that's also something that we're seeing emerge around some of these agentic tasks, the rise of these agentic setups. And there's even companies that are now. you know, very big, you know, that are kind of Arising that are purely focused on creating these benchmarks and running them.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-118" }, { "index": 119, "time": "49:29", "seconds": 2969, "speaker": "Conor Bronsdon", "text": "(Sponsor read: WisdomAI) Any frontier model can query your data. But can you trust its answers? This episode's sponsors, WisdomAI, put that to the test in their latest benchmark study. Alone, Claude Code topped out at 73% accuracy. WisdomAI closed the gap, delivering accurate answers 3x faster at a third of the cost. I sat down with Kapil Chhabra, WisdomAI's co-founder and CPO for episode 75, and he explained the analytics architecture that makes these kind of results possible. Check it out for a full deep dive, plus get the blueprint for trusted AI analytics at wisdom.ai slash benchmark so you can deliver accurate answers on Claude, ChatGPT, or wherever your business gets their insights. Download your copy at wisdom.ai slash benchmark.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-119" }, { "index": 120, "time": "50:21", "seconds": 3021, "speaker": "Conor Bronsdon", "text": "Yeah, we had a really interesting conversation about open benchmarks as well with uh Snorkel earlier this year where they have been funding a bunch of open benchmarks. Um which has been I think a really interesting element of this year because To your point, we're seeing a lot of benchmaxxing. We're seeing Uh A variety of benchmarks fall Mm. of rapidly changing space. And Yeah. can be challenging for some folks. to understand what does success actually look like. A recent example of this is Google hasn't officially released as of recording, this may change when we actually publish this episode, their new Gemini 4 Argon, but they released their benchmarks for it and they're like, oh, we're crushing all these benchmarks.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-120" }, { "index": 121, "time": "50:27", "seconds": 3027, "speaker": "Saahil Jain", "text": "Yeah. That's a good question. I think it's similar in search. I think there are A lot of benchmarks that don't correlate to real world performance. So this is something that we have noticed. And there's also different types of benchmarks. It depends a little bit about what you want to focus on. I'll give you an example of how we're evolving. A lot of these public public benchmarks are static. What that essentially means is they're not changing. Um over time. And this allows you to essentially, especially with search, essentially memorize the answer.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-121" }, { "index": 122, "time": "51:05", "seconds": 3065, "speaker": "Conor Bronsdon", "text": "Look at how well we did on Terminal Bench. But there's a lot of doubt there because there's been some experience as far as real use that in the past Google models. Have um tended to be stronger on benchmarks than they were in real-world tasks. How do you think about that from a a search perspective going forward? And what should actually be What should success look like in this space?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-122" }, { "index": 123, "time": "52:00", "seconds": 3120, "speaker": "Saahil Jain", "text": "And that can allow companies to do really well on static benchmarks. Um without necessarily Having generalization capabilities to questions outside of the benchmark, and that obviously would lead to a distribution shift in performance. So what we've been thinking a lot about is dynamic benchmarks. So internally, we've been working on Live QA. Something that we'll release as well, which is a benchmark that changes over time. Then we also worked on vertical RTK with another partner. Um which is a benchmark that changes even more often. It asks questions and what's the ground truth.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-123" }, { "index": 124, "time": "52:26", "seconds": 3146, "speaker": "Saahil Jain", "text": "Yeah. uh procured you know a couple hours later as opposed to even days later. So freshness becomes really important. So that's one piece when it comes to benchmarks is making sure that You're focusing on benchmarks that are somewhat aligned with your capabilities. For us, we care a lot about freshness because we think that's one of the core capabilities of a search API. And we wanna use benchmarks to help us improve. that are forcing us to make sure that our APIs are fresh rather than just being really good at static benchmarks.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-124" }, { "index": 125, "time": "52:52", "seconds": 3172, "speaker": "Saahil Jain", "text": "Static benchmarks are also useful just make sure that the reasoning capabilities of the models are working well with your search tool. So they are useful to some degree as well. But I do think that it is. quite possible to overfit on them. And then to be less useful. And then it comes down to, okay, so that's one area is you can use certain benchmarks. to make yourself better, and you can be targeted in terms of the benchmarks you choose to be good on. And then it comes down to, for actual customers, what do we think helps?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-125" }, { "index": 126, "time": "53:17", "seconds": 3197, "speaker": "Saahil Jain", "text": "At the end of the day, a lot of it comes down to testing. So I think the best validation is customers using you at scale. That's that that's what matters. You know, I think that's very hard to game. If a customer is using you and, you know, using you and for tens of millions or even a billion queries a month. etcetera. Like that is something that Um cannot be gamed because there's, you know, an actual customer on the other side who's feeling the impact. of everything that the agent does, including the tools that it uses.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-126" }, { "index": 127, "time": "53:41", "seconds": 3221, "speaker": "Saahil Jain", "text": "So I think that At the end of the day. The customer success is the most important. And what we often do within You.com is we focus on partnering really closely with our companies, and we have an AI evals team that focuses on doing custom evals. for companies. And I think we'll see more and more companies go in that direction, or they start building company specific evals. Because at the end of the day, that's what matters most of the company. Like a benchmark is simply a way of simulating what a company would care about.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-127" }, { "index": 128, "time": "54:06", "seconds": 3246, "speaker": "Saahil Jain", "text": "Why not just go directly to the company and work with them to create an eval set? Let's skip that whole s that whole step. So I think that's what we're seeing. a lot of the future of benchmarking as these public benchmarks become more saturated.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-128" }, { "index": 129, "time": "54:19", "seconds": 3259, "speaker": "Conor Bronsdon", "text": "Oh, I've really enjoyed this conversation. Do you have any closing thoughts for the audience as we wrap up?", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-129" }, { "index": 130, "time": "54:24", "seconds": 3264, "speaker": "Saahil Jain", "text": "Yeah, I think my main closing thought would be I think that there's so much that's changing day to day. I feel like it's Important to keep an open mind, and that's what we're doing as well. Is you know, we're keeping an open mind. I think we're revising our philosophy day by day. You know, I give you an example of some ways in which I've revised my philosophy over the last couple of months. before I kind of you know, I did have the opinion that harnesses would be completely rendered irrelevant.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-130" }, { "index": 131, "time": "54:46", "seconds": 3286, "speaker": "Saahil Jain", "text": "By these model intelligence capabilities rapidly improving. But I think with the rise of open source models and model diversity becoming more important now than it was one or two years ago. because there's actually good open source models. I think it actually is reviving a lot of harnesses to a degree if they can properly coordinate across all these models out there. And I think the fact that even Codex is doing this. That which I learned today from you is a good validation of this too. So I think it's important to keep.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-131" }, { "index": 132, "time": "55:13", "seconds": 3313, "speaker": "Saahil Jain", "text": "And open mind and keep shifting your thesis as new information comes about. And I'm sure this is not the last time I'll. shift my opinion on things. Maybe if we talk, you know, maybe six months or a year from now, I'm sure I'll be saying very different things than I'm saying right now. Um but yeah, that's that's that's that's what I think is the nature of the game right now is Things are changing so fast and it's important to keep an open mind.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-132" }, { "index": 133, "time": "55:26", "seconds": 3326, "speaker": "Conor Bronsdon", "text": "Same here, man. 100% agree. I really appreciate you coming on and sharing your insights with the audience. It's been a great discussion. And I think, honestly, we'd love to have you back at some point because we clearly just scratched the surface of your knowledge. And to your point. I think we're all changing our opinions very rapidly sometimes. in this space because of everything that's going on. And I mean, honestly, it's why I named this show Chain of Thought is You can probably look back at things that I was saying in episode one that are completely wrong for the time we're now at episode 77 or whatever.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-133" }, { "index": 134, "time": "56:04", "seconds": 3364, "speaker": "Conor Bronsdon", "text": "And we all need to be thinking through continually how we are addressing this industry because uh the capabilities are changing so fast. Uh the models are improving rapidly in ways we don't always expect. And there are so many intelligent people that are working on these problems in different angles that there are always going to be surprises and things for us to learn from. Honestly. That's most of the fun to me. So Saahil, thanks so much for joining me today. It's been fantastic chatting with you. I really appreciate you", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-134" }, { "index": 135, "time": "56:32", "seconds": 3392, "speaker": "Conor Bronsdon", "text": "bringing your insights. And for anyone who listened to this conversation and enjoyed it, maybe drop a comment below and let us know what you thought. Let us know what questions you had that we didn't get to. and we'll make sure we address it next time. Saahil, great to see you. Thanks again for coming on.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-135" }, { "index": 136, "time": "56:45", "seconds": 3405, "speaker": "Saahil Jain", "text": "Yeah.", "url": "https://chainofthought.show/podcast/81-agents-are-overtaking-humans-in-search-why-search-must-be-rebuilt/#t-136" } ], "usage": { "attribution": "Chain of Thought — chainofthought.show", "canonical": "Every record carries canonicalUrl. Cite that page, not this endpoint: it is the authoritative rendering and the one that gets corrected.", "transcripts": "Machine-generated (ASR), with entity corrections applied during sync. Transcripts may contain errors; verify quotations against the recording.", "license": "CC-BY-4.0", "audio": "Episode audio and video are not licensed: all rights reserved. The audio and video URLs are links only; link to the episode page instead of copying the media.", "redistribution": "Transcript text and episode metadata are licensed CC BY 4.0: credit Chain of Thought, link the episode page and the license, and say if you changed the text. The open dataset at https://huggingface.co/datasets/conorbronsdon/chain-of-thought-podcast-transcripts is a snapshot of these records, refreshed periodically." } }