Ask an agent to find the cheapest delivery option for a sofa from three furniture stores. None of them has a public API, so the agent opens each site, searches, adds the item to the cart, enters a zip code and reads the shipping options, then reports back. That is a browser agent. It is a web-specific case of computer use, which covers any desktop interface.
Browser agents see a page through a screenshot, the page’s structure, or both, and choose the next action in a loop. That makes them flexible but slow and error-prone next to an API call. Pages change layout, load slowly, hide content behind pop-ups and sometimes block automation. Research benchmarks show how far the technology has come and how far it has to go: WebArena’s best 2023 agent finished about one task in seven that humans completed most of the time.
In episode 63 at 16:41, Yutori’s Dhruv Batra argues that only the biggest sites will build agent-friendly APIs, while the long tail of the web has to become usable as it is. Browser agents, which Yutori builds, are his answer: if a person can look at the screen and know which button to click, an agent can learn to do the same, then run thousands of times in parallel (18:42). He also notes that accessibility trees and web standards are applied inconsistently, so what the human sees is the real source of truth. That is a founder’s case for his market.
Anything a browser agent reads can try to steer it, so web pages are a route for prompt injection. Run browser agents in a sandbox, give them the least privilege they need, and require a person to confirm purchases, logins and other actions that can’t be undone (human in the loop).
Hear it from the guest
“If you as a human can look at a screenshot and look at a browser screen and know which button to click, that is the operation that a browser agent has to do.”
Quotes lightly edited to remove filler words.
Sources
- Zhou et al.: WebArena, a realistic web environment for building autonomous agents (2023) — Builds working websites across e-commerce, forums, software development and content management; its best GPT-4 agent reached a 14.41% end-to-end task success rate against 78.24% for humans; checked October 24, 2026.
- He et al.: WebVoyager, an end-to-end web agent with large multimodal models (2024) — A multimodal web agent that completes tasks on 15 real websites from screenshots and page content, reaching a 59.1% task success rate on its benchmark; checked October 24, 2026.
- OpenAI API docs: Computer use — Documents letting a model operate browser and desktop interfaces through screenshots and actions; checked October 24, 2026.
