In detail
1. The constraint moved downstream
Graphite co-founder Greg Foster traces the arc from autocomplete to chat to agents in the terminal, and then to the shape people have started calling background agents: you message one on Slack or GitHub, wait a couple of hours, and a pull request comes back. His framing is that this is the self-driving version rather than highway autopilot. Once generation runs unattended, the queue forms at review, and review was already the slow step.
2. Human review was unreliable before agents arrived
Foster is blunt that review quality has always depended on who is reviewing and how much attention they are paying, with “LGTM” as the failure mode. Agents make that worse from both ends: there are more pull requests, and models are not concise, so the diffs are longer. A process that already leaked at human volume does not hold at agent volume.
3. Tests become the surface you actually review
AMD’s Anush Elangovan describes running 10 to 12 main agents across his machines (7:46), and says his token count had reached about 6.5 billion, up from the 4.2 billion a week he cited when he and Conor last spoke (10:54). His rule predates agentic coding: 80% of your time goes to tests, and the test framework exists before the first line of code (14:12). In his workflow the harness is the gate a change has to pass (18:39). Passing it is evidence about the behaviors the harness checks, not certification of every property of the change. It is the move evaluation already made for models: you stop reading outputs one at a time and build something that checks them.
Update, October 3, 2026: At 5:59 in episode 74, Laurie Voss discusses the next two to five years and estimates that the best AI code-review tools Voss has seen have about an 80% hit rate. Voss calls that good but insufficient for shipping and describes a continuing role for senior, AI-assisted human code review focused on the remaining problems. This is Voss’s assessment at the time of the interview, rather than a measured benchmark across review tools.
4. Review the boundary, not every line
Elangovan’s agents run on a geo-distributed test bed on his own private network (9:15), and he calls testing “the gate of your sandbox” (18:39). That is the practical shape of the human-in-the-loop checkpoint when volume makes per-line reading impossible: pick the boundary where a mistake becomes expensive, and put the human there. Everything inside the boundary is the harness’s job.
What each kind of evidence covers
No single check settles whether a change is ready. NIST’s Secure Software Development Framework lists reviewing or analyzing code and testing executable code as separate practices (NIST SP 800-218, SSDF 1.1). This table is our summary of what each kind of evidence can and cannot tell a reviewer.
| Evidence | What it helps establish | What still needs checking |
|---|---|---|
| Functional and integration tests | Specified behavior on exercised cases | Whether expectations are correct and important cases are missing |
| Source, dependency and secret analysis | Issues covered by the tools and rules used | False alarms, missed issues and context outside those rules |
| AI review comments | Candidate defects or questions to investigate | Confirmation against the actual diff and requirements |
| Human review and rollout evidence | Design intent, operational fit and observed effects | Ownership and recovery if behavior differs after release |
Passing tests are not a permission boundary either. Run generated code and test commands in an environment restricted to what the task needs.
Why it matters
The instinct when agent output floods the queue is to read faster. That does not scale, and it converts review into a rubber stamp with extra steps. Decide instead what your tests must prove and where a human sign-off changes the outcome, then let the agents fill the space between.
A review sequence for an agent-written PR
This is our suggested workflow, drawing on the distinction above between tests, review and merge ownership. It is not a claim that every guest uses these five steps.
- Define the requested behavior. Name the outcome, constraints and parts of the system the change may affect. A passing test for a different task does not establish completion.
- Inspect the checks as well as the patch. Read changed tests, fixtures and configuration. Confirm that a test still rejects a known incorrect result in an isolated case; an agent can make a test pass by weakening what it checks.
- Use another reviewer for a focused question. Ask for defects with file-level evidence, then reproduce each finding. A second model’s agreement does not replace source inspection or testing.
- Verify the user-visible result. Run the relevant tests and exercise the affected flow. Check required side effects in the system of record, including actions that must not occur. Record what the check actually covered.
- Keep the merge decision with its owner. Review the final commit and unresolved findings before approval. If the patch changes after review, recheck the affected behavior and obtain the reviews your release policy requires.
Keep the review evidence tied to the commit being considered. If the result is still uncertain, name the missing check instead of converting an agent’s “done” into a release decision.
A worked example: green CI still needs review
Conor describes his workflow in his October 2026 newsletter essay on reviewing AI code. On pull request #383 for avoid-ai-writing, his open-source writing skill, CI was green. Qodo and Devin still found cases the detector missed. The requested repairs included verb uses such as “a tool that features…” and “iOS 18 features…”. Review also identified a pre-existing highlighting problem: a word used as a noun was highlighted because a later sentence used the same word as a verb.
The final review record separates three further static model reviews, from different families, from the lead model’s executed checks. The lead ran the affected tests, including forty new paired test cases, and confirmed seven passing GitHub checks. The record also disclosed earlier local timing failures and the baseline reproductions. This is one repository’s review, not a measured comparison of review methods.
Disclosure: Episodes 15 and 40 aired in season two, which Galileo sponsored while Conor worked there; both episodes’ show notes carry a Galileo block. Episode 56 was sponsored by Galileo, the Season 3 presenting sponsor (read at 2:38). Inngest, Svix, Walrus Memory and G2i sponsor season four, including episode 74 (4:38).
Hear it from the guest
“Maybe like no human has ever read this code before. There's like a bot that just generated it. And now I'm the first person ever reading it.”
“Testing is the code review and testing is kind of the gate of your sandbox. If you have a complaint about anything in the box, fix the sandbox, right?”
“I would say the best code review AI tools that I've seen have like an 80% hit rate.”
“Everything is so perfectly commented in your code, I'm like, you didn't write this.”
Quotes lightly edited to remove filler words.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
Concepts in this explainer
AI Code GenerationAI Code ReviewAI SlopProduct engineerSpec-Driven DevelopmentTechnical DebtUnit TestHuman in the Loop