Skip to content

AI, decoded

Who reviews the code when AI writes most of it?

A human still owns the merge, but the review cannot stay a line-by-line read of the diff. Generation stopped being the constraint the moment background agents could open pull requests unattended, and the teams keeping up moved the check that decides into the test harness. AMD's Anush Elangovan told Chain of Thought that testing is the code review: the test harness sets hard requirements agents must meet, giving a 50-microsecond dispatch time in his GPU simulator as an example.

· Updated · Chain of Thought

Level 3: Leading with AI · 3.6 Teams and skills

AI CodingAI EngineeringAI Evaluation & Reliability

  • 4 episodes
  • 4 guest clips
  • 8 glossary terms
A struck-through LGTM rubber stamp beside a tall pile of pull requests, over AMD's Anush Elangovan's line from episode 56: testing is the code review.

In detail

1. The constraint moved downstream

Graphite co-founder Greg Foster traces the arc from autocomplete to chat to agents in the terminal, and then to the shape people have started calling background agents: you message one on Slack or GitHub, wait a couple of hours, and a pull request comes back. His framing is that this is the self-driving version rather than highway autopilot. Once generation runs unattended, the queue forms at review, and review was already the slow step.

2. Human review was unreliable before agents arrived

Foster is blunt that review quality has always depended on who is reviewing and how much attention they are paying, with “LGTM” as the failure mode. Agents make that worse from both ends: there are more pull requests, and models are not concise, so the diffs are longer. A process that already leaked at human volume does not hold at agent volume.

3. Tests become the surface you actually review

AMD’s Anush Elangovan describes running 10 to 12 main agents across his machines (7:46), and says his token count had reached about 6.5 billion, up from the 4.2 billion a week he cited when he and Conor last spoke (10:54). His rule predates agentic coding: 80% of your time goes to tests, and the test framework exists before the first line of code (14:12). In his workflow the harness is the gate a change has to pass (18:39). Passing it is evidence about the behaviors the harness checks, not certification of every property of the change. It is the move evaluation already made for models: you stop reading outputs one at a time and build something that checks them.

Update, October 3, 2026: At 5:59 in episode 74, Laurie Voss discusses the next two to five years and estimates that the best AI code-review tools Voss has seen have about an 80% hit rate. Voss calls that good but insufficient for shipping and describes a continuing role for senior, AI-assisted human code review focused on the remaining problems. This is Voss’s assessment at the time of the interview, rather than a measured benchmark across review tools.

4. Review the boundary, not every line

Elangovan’s agents run on a geo-distributed test bed on his own private network (9:15), and he calls testing “the gate of your sandbox” (18:39). That is the practical shape of the human-in-the-loop checkpoint when volume makes per-line reading impossible: pick the boundary where a mistake becomes expensive, and put the human there. Everything inside the boundary is the harness’s job.

What each kind of evidence covers

No single check settles whether a change is ready. NIST’s Secure Software Development Framework lists reviewing or analyzing code and testing executable code as separate practices (NIST SP 800-218, SSDF 1.1). This table is our summary of what each kind of evidence can and cannot tell a reviewer.

EvidenceWhat it helps establishWhat still needs checking
Functional and integration testsSpecified behavior on exercised casesWhether expectations are correct and important cases are missing
Source, dependency and secret analysisIssues covered by the tools and rules usedFalse alarms, missed issues and context outside those rules
AI review commentsCandidate defects or questions to investigateConfirmation against the actual diff and requirements
Human review and rollout evidenceDesign intent, operational fit and observed effectsOwnership and recovery if behavior differs after release

Passing tests are not a permission boundary either. Run generated code and test commands in an environment restricted to what the task needs.

Why it matters

The instinct when agent output floods the queue is to read faster. That does not scale, and it converts review into a rubber stamp with extra steps. Decide instead what your tests must prove and where a human sign-off changes the outcome, then let the agents fill the space between.

A review sequence for an agent-written PR

This is our suggested workflow, drawing on the distinction above between tests, review and merge ownership. It is not a claim that every guest uses these five steps.

  1. Define the requested behavior. Name the outcome, constraints and parts of the system the change may affect. A passing test for a different task does not establish completion.
  2. Inspect the checks as well as the patch. Read changed tests, fixtures and configuration. Confirm that a test still rejects a known incorrect result in an isolated case; an agent can make a test pass by weakening what it checks.
  3. Use another reviewer for a focused question. Ask for defects with file-level evidence, then reproduce each finding. A second model’s agreement does not replace source inspection or testing.
  4. Verify the user-visible result. Run the relevant tests and exercise the affected flow. Check required side effects in the system of record, including actions that must not occur. Record what the check actually covered.
  5. Keep the merge decision with its owner. Review the final commit and unresolved findings before approval. If the patch changes after review, recheck the affected behavior and obtain the reviews your release policy requires.

Keep the review evidence tied to the commit being considered. If the result is still uncertain, name the missing check instead of converting an agent’s “done” into a release decision.

A worked example: green CI still needs review

Conor describes his workflow in his October 2026 newsletter essay on reviewing AI code. On pull request #383 for avoid-ai-writing, his open-source writing skill, CI was green. Qodo and Devin still found cases the detector missed. The requested repairs included verb uses such as “a tool that features…” and “iOS 18 features…”. Review also identified a pre-existing highlighting problem: a word used as a noun was highlighted because a later sentence used the same word as a verb.

The final review record separates three further static model reviews, from different families, from the lead model’s executed checks. The lead ran the affected tests, including forty new paired test cases, and confirmed seven passing GitHub checks. The record also disclosed earlier local timing failures and the baseline reproductions. This is one repository’s review, not a measured comparison of review methods.

Disclosure: Episodes 15 and 40 aired in season two, which Galileo sponsored while Conor worked there; both episodes’ show notes carry a Galileo block. Episode 56 was sponsored by Galileo, the Season 3 presenting sponsor (read at 2:38). Inngest, Svix, Walrus Memory and G2i sponsor season four, including episode 74 (4:38).

Where code review moves when agents write the code Before: agents send more pull requests with longer diffs into one human reading line by line, so the queue forms at review and it slides into LGTM. After: agents work inside an enclosure whose wall is the test harness. Passing it is evidence about the behaviors it checks, not certification of every property. Code and dependency scans and AI review comments add evidence, and AI findings are hypotheses to confirm against the actual change. A human still owns the merge at the boundary where a mistake becomes expensive, reading the changed tests and reviewing the final commit. Where the review moves Generation runs unattended. Put the check where a mistake becomes expensive. BEFORE · READ EVERY LINE agent agent agent LGTM More pull requests, longer diffs, one reader. The queue forms at review, and review slides into a rubber stamp. AFTER · REVIEW THE BOUNDARY test harness agent agent agent agent Passing shows only what the harness checks. + scans, AI findings: check the diff Human owns the merge reads changed tests, reviews final commit Review what enters and leaves the enclosure, where a mistake becomes expensive. Decide what the tests must prove, then where a human sign-off changes the outcome. Graphite’s Greg Foster, ep 40: message a background agent, wait a couple of hours, and a pull request comes back. AMD’s Anush Elangovan, ep 56: 80% of your time on tests, and the test framework exists before the first line of code.
Once generation runs unattended, let the test harness carry the routine checking and keep a human on the merge, where a mistake becomes expensive. The 80% rule is Anush Elangovan’s, from episode 56. Download the image

Hear it from the guest

“Maybe like no human has ever read this code before. There's like a bot that just generated it. And now I'm the first person ever reading it.”
“Testing is the code review and testing is kind of the gate of your sandbox. If you have a complaint about anything in the box, fix the sandbox, right?”
Watch on YouTube
“I would say the best code review AI tools that I've seen have like an 80% hit rate.”
Watch on YouTube
“Everything is so perfectly commented in your code, I'm like, you didn't write this.”

Quotes lightly edited to remove filler words.

From the conversation

This explainer is drawn from these episodes — each carries its full transcript.

Concepts in this explainer

AI Code GenerationAI Code ReviewAI SlopProduct engineerSpec-Driven DevelopmentTechnical DebtUnit TestHuman in the Loop