Skip to content

Vikram Chatterji

Co-founder and CEO of Galileo

  • 5 episodes on Chain of Thought
  • 9 topics covered
  • 5 full transcripts
  • Nov 2024 first on the show
Vikram Chatterji

Vikram Chatterji is the co-founder and CEO of Galileo, an AI reliability and evaluation company, and a recurring co-host of Chain of Thought in the show’s first season. Across several episodes he broke down eval engineering, the ROI of enterprise AI, and why evaluations need to become infrastructure rather than one-off tests.

Cover art for Explaining Eval Engineering | Galileo's Vikram Chatterji

Latest episode with Vikram Chatterji · EP 46

Explaining Eval Engineering | Galileo's Vikram Chatterji

Vikram Chatterji, Galileo ·

Show notes & transcript → All listening options →

Episodes

What the conversations cover

These notes summarize the episodes featuring Vikram Chatterji, including other speakers in panel conversations. Follow a source link to read the discussion in context.

EP 46 Explaining Eval Engineering | Galileo's Vikram Chatterji

  • Generic LLM-as-a-judge metrics plateau around 70% accuracy — unshippable for customer-facing AI. A GPT-5 judge is roughly 70% accurate, ~$1.25 per million tokens, and 2-3 seconds of latency. At 70%, three in ten interactions fail and you often can’t tell, so that’s just not deployable for anything particularly important. Episode takeaways
  • Galileo’s path off the plateau is three steps: auto-author the judge prompt from a subject-matter expert’s plain language (“Auto-Gen,” ~75%), close the “last-mile measurement” gap with continuous learning through human feedback (“Auto-Tune”/CLHF — a handful of feedback points gets it to ~95%), then distill it into a fine-tuned small language model (the Luna evaluation models) for low latency and cost at scale. Episode takeaways
  • Evals are already most of the job. When Vikram’s team asks serious AI builders — AI SRE orgs, billing agents for sales and banks — what their day looks like, roughly 70% of the time is just evals, mostly tweaking judge prompts. Because of that, he predicts “eval engineer” becomes a standard role, the way “prompt engineer” once was. Episode takeaways

EP 25 The 2025 AI Shift: From Chat to Task Completion & Reliable Action | Galileo Founders

  • Vikram Chatterji frames 2025’s defining shift as generative AI moving from chat completion toward task completion and action. He argues this elevates the enterprise conversation beyond building chatbots to automating tasks, with the goal of higher ROI on OpEx and CapEx — though he concedes “agents” is “an overblown term.” Episode takeaways
  • Vikram describes a “gold rush” among middleware providers racing to build orchestration and frameworks — citing OpenAI’s Agents SDK, Anthropic’s MCP, and Google’s A2A. He’s blunt that some “don’t mean anything” and are “shallow libraries,” but reads the collective effort as the system falling into place to make task-completion apps real. Episode takeaways
  • Atin Sanyal pitches evaluation agents as composable functions that sit where the developer works — for example an eval agent installed as a tool inside Cursor that automatically fixes potential issues. He argues eval is expanding into a broader AI reliability story rather than the point-in-time activity it used to be, with MCP servers standardizing communication around the LLM. Episode takeaways

EP 13 How Engineering Leaders Can Future-Proof Their AI Strategy | Vikram Chatterji on Dev Interrupted

  • Vikram Chatterji argues AI is “another tool in your arsenal,” not a mandate. The right first question isn’t “how do we use AI” but “what’s your use case, and based on that, is this a good fit for your use case?” — and engineering leaders need the machinery to push back and say “maybe I don’t need to use AI at all.” Episode takeaways
  • Risk tolerance should set the pace. Chatterji contrasts a bank — where a misfiring consumer AI can “derail your entire bank’s reputation” because it deals with people’s money — against a DoorDash or Instacart, where a chatbot slip is “dealing with hungry people,” not bank accounts. Enterprises go crawl-walk-run; digital natives move fast and break things. Episode takeaways
  • Even fast-moving teams need forecasting before they build. Chatterji’s sequence: forecast how many use cases (10, 15), provision enough compute (he name-checks A100 GPUs) so engineers aren’t asking “where’s my GPU at,” invest in tooling to cut compute cost, then add eval guardrails for an “AI CI/CD process” before letting teams loose. Episode takeaways

EP 4 Why Most Enterprise AI Projects Fail to Show ROI | HP, ServiceNow & Accenture

  • The enterprise “ROI of AI” panic is a correction, not a dead end. Vikram Chatterji frames it as a “trough of disillusionment”: enterprises that overpaid for expensive frontier models 8–10 months ago, spending “maybe 50 million dollars” on a couple of applications, are now whiplashed when the returns lag — but “we’re still in the first innings.” Episode takeaways
  • ROI from enterprise AI rarely shows up as new revenue. Chatterji notes there is no “net new AI-powered bank” being built — the value is cost savings and operational efficiency, measured over six to eight months by comparing human-capital cost before and after and how many humans-in-the-loop a workflow still needs. Episode takeaways
  • Trust gates deployment at HP. Alex Klug: “the worst thing that can happen is we are in the news about having deployed a solution that has steered our customers wrong.” His CDO puts it bluntly — “you guys are building rocket ships, I want a braking system” — and that missing explainability is what keeps CIOs from moving past one or two pilots. Episode takeaways

EP 3 GenAI Predictions for 2025 | Databricks & Cohere

  • Vikram Chatterji predicted that by the end of 2025 open-source LLMs would catch up to the best closed-source models — naming Anthropic’s Claude models which are “super expensive” and which he almost thinks of as “concept models” showing “the art of the possible” — for large context windows on a majority of tasks, citing Galileo’s Hallucination Index as evidence open source was closing the gap fast. Episode takeaways
  • Chatterji predicted 5-to-10 of what were dismissed in 2023 as “wrapper apps” would each hit $100M in ARR by the end of 2025, arguing AI breaks the old mobile-app adoption curve where even DoorDash or WhatsApp took a while to reach $100M in revenue. Episode takeaways
  • Yash Sheth called 2025 “an era of toolification,” and Wiley agreed: rather than “boil the ocean” with one single my-company-GPT.com, teams deploy narrow, purpose-built tools and smaller agents and orchestrate them together — a model tuned for a narrow capability beats one giant bot, and function-calling usage now signals a team’s AI maturity. Episode takeaways

In their words

That's the research that we've that we've been doing for the last couple of years. So continuous learning through human feedback basically means that now you add that natural language feedback, like I said, and we auto tune the LLM as a judge. And now very quickly with four or five pieces of feedback, you can get it to a fairly high degree of accuracy.

Vikram Chatterji Explaining Eval Engineering | Galileo's Vikram Chatterji · 13:30

And that's kind of where these platforms like Galileo come in, and it's gonna get more and more important for us to be able to think of solving their problem in a very holistic way.

Vikram Chatterji The 2025 AI Shift: From Chat to Task Completion & Reliable Action | Galileo Founders · 29:01

Transcript excerpts; ellipses mark omitted words. Some automatically selected quotes have filler words removed.