My Robot Army: How I review AI code

The essay describes how Conor Bronsdon reviews agent-written code without reading most diffs himself. His part is setting the task and its limits. Agents write the code and gather proof that it works, an oversight model audits that proof, and Conor steps in for exceptions and judgment calls. He excepts critical infrastructure and life-threatening code.

The loop starts with cheap checks: lint, types, tests and build. Code that passes goes to review by a different model family, to reduce shared blind spots. The essay cites research on the limits of that step. Across more than 350 models, two models that both got a question wrong on one leaderboard gave the same wrong answer 60% of the time (Kim et al.), and models in multi-agent debate often abandon correct answers to agree with peers (Wynn et al.). Examples come from his avoid-ai-writing project, where the review tool Qodo caught tests that could print “FAILED” and still pass.

The essay draws on Anush Elangovan’s test-first workflow in episode 56, Kimberly Logan on portable memory in episode 77 and Laurie Voss on product engineering in episode 74.

Key points

• Run cheap automated checks first, so known failures are fixed before tokens are spent on review.
• Agreement between reviewers has to rest on evidence. A useful finding names the input, the expected behavior, the actual behavior and the code responsible.
• Reproduce a finding before it drives a merge decision. Some reviewer claims described old bugs or rested on bad tests.
• Review the exact commit being merged. Approval of an earlier version does not cover later fixes.
• Human judgment stays for product direction, architecture, permissions and unresolved tradeoffs.

Disclosure: The essay was sponsored by Walrus Memory. In it, Conor discloses that he has consulted for CoralBricks, a model-access provider he uses, and that he receives free automated reviews from Qodo through its program for open source projects. Walrus Memory also sponsored episode 77, whose guest works at Walrus (disclosure at 3:24). Episode 56 was sponsored by Galileo, the Season 3 presenting sponsor (read at 2:38). Inngest, Svix, Walrus Memory and G2i sponsor season four, including episode 74 (4:38), and episode 76 credits Svix, Walrus Memory and Inngest as Season 4 presenting sponsors (Svix read at 14:55).