The Infrastructure Behind Agents That Run for Months

Conor Bronsdon 1:13 I am super excited for this conversation because I think there is this sense that agents are changing the world, but the specifics of them can be very nitty-gritty. So Tony, you built Inngest before the agent wave started, doing background jobs and serverless functions. What did agents change around your approach, and how are you thinking about the future?

Tony Holdstock-Brown 1:32 Yeah, that's super interesting and a really, really, really good call out, because I think Inngest started, like you say, before agents and it was all about workflows, a series of steps that run, making them durable, making it work so that you didn't really need to think about the infrastructure as you were writing your functions. And it turns out that it translated really, really well to agents because agents are just a series of steps, and it might be a series of steps that recursively calls the same LLM.

Tony Holdstock-Brown 1:56 It might be a series of steps that calls an LLM, delegates to sub-agents, which then run a series of tasks and propagate that data back to the parent. But it's essentially the same thing. So the mental model that we had prior to agents almost mapped pretty directly. Yeah. And it meant that the transition was relatively smooth. We didn't really need to re-architect or build too many different things. Um, because, uh, again, like, it's essentially one long chain of a function. Breaking that up into a series of steps was really good for us because then you were able to see.

Tony Holdstock-Brown 2:30 Sort of deterministically what your agents are doing at any point in time with the traces that you get. And if you wanted to replay an agent's trajectory from, say, step 20 of a thousand step function, it's totally possible. But the interesting thing is, like, the way that people used us massively changed. Um, and it turns out that, like, obviously if you build something for workflows, background stuff, it's typically really async. Uh, if people are building agents, the first wave of agents, which are prior to the long run agent stuff we were talking about.

Tony Holdstock-Brown 2:59 Um, very, very, very synchronous, very chat focused. Um, less like, ask an agent to do something, it has its own trajectory, or figure it out, because the models weren't as good back then. And so you'd be coaching the agent a lot. And that was very, very synchronous. So speed and latency was really important. We had to put in some changes to make sure that latency was really, really low, to make sure that the workflows are running as fast as possible. Um, but apart from that, like, the mapping carries over pretty directly.

Tony Holdstock-Brown 3:27 And this is maybe one observation as well, I think, along the way of building with AI. There's a lot of new patterns. And interestingly, a lot of old patterns that still map directly to building with AI that people may have forgotten. Which is really, really cool to see, cool to re-explore.

Conor Bronsdon 3:44 Good systems engineering is still good systems engineering.

Tony Holdstock-Brown 3:47 Yeah, yeah, exactly that. Exactly that. So, uh, we were, we were pretty lucky.

Conor Bronsdon 3:53 I, I think it more is that you built the right primitives in advance, and now you're continuing to be able to build on top of that. I had a similar conversation with the chief product officer of Slack recently, Jaime DeLanghe. And Jaime and I talked about the fact that Slack as a primitive for context storage is really excellent for coding agents. And that's how this coding channel thing has spun up for them. And why Anthropic ships more than 50% of their code in Slack. And I think we're seeing that same thing with platforms like Inngest, where because you have set up to understand and ingest data the way that people need to in their day-to-day lives.

Conor Bronsdon 4:29 When you see an agent fleet start to take that on as well, you're prepared and just simply need to scale up what you're able to do with it. Crucially though, I'd want to define one term you're using because I think it's really interesting and maybe it'll help level set the conversation. Long running agents. Where's the line?

Tony Holdstock-Brown 4:47 A really, really, really interesting question, and I think really easy to talk about two extremes. You've got your sort of Google-esque, type in something in the search bar and now Google will give you an AI response. Absolutely not running, single shot. The stuff that you've been doing since the early days of, like, GPT-3. And then recently, we're recording this and yesterday it was Anthropic's Duet that was announced. Um, you've got a Grokbot, Duet, Muse and so on, and Dots, Instinct, yeah, Instinct like being one of the early ones, but it was like very invite only.

Conor Bronsdon 5:16 Instinct, there's so many of them.

Tony Holdstock-Brown 5:22 Um, and I say earlier, that was like two months ago. Um, these are like personal agents that run twenty four seven. Almost like OpenClaw. Um, and these are like purely long-running agents that maintain state, orchestrate sub-agents. Basically go through, figure out which tasks you want done, whether that's your feedback or things you've set up for it. And, uh, the definition of a long-running agent is they will live for months, building up their own context. Mm-hmm. This is super interesting. Because think more and more, especially over the last month, we've seen that.

Tony Holdstock-Brown 5:56 Um, and the power is crazy. One of our folks internally uses stuff like this to do a lot of his work. Um, and it's incredibly interesting, just how fast you can go with this.

Conor Bronsdon 6:08 Let's talk a bit about how long-running agents are being leveraged today, because I spent a lot of time in the last year engineering my context system so that agents could hand off to each other easily. And I think that a lot of engineers spent time doing this. And now we have agents that are seemingly able to handle so much of this process for us. What does it look like to use long-running agents today when it comes to the workplace versus consumer side?

Tony Holdstock-Brown 6:32 I th- I think like a few interesting th- threads to pull. Number one, yeah, I think we did the same thing. Originally, way back when, we had like a deterministic layer over agents. Turns out bitter lesson gets us all, you know, like agents just get better. That's crazy, and that's super interesting. For us, I think like we see a few different things in terms of customer base, and then I'll talk about architecture. And then also just the market. You've got people that are using agents internally, personally, for their own sort of like personal projects, or their own useful life.

Tony Holdstock-Brown 7:00 And we've got a bunch of companies that are building long running agents on top of Inngest. And you've also got like Instinct, Muse, that sort of stuff that's like come out and really kind of change the game for a lot of people personally. And I'm gonna, I'm gonna position this as a difference to what used to happen with Claude locally. In Claude Code. Claude Code. You would run things on your one machine and you could ask it to spin up a sub-agent. Pretty cool. And you'd be like, use as many subagents as you want to review.

Tony Holdstock-Brown 7:26 And it would do everything on your machine. Pretty sick. But if you're building something like Dots or you want to build something like Duet, you end up with the central orchestrating system that's doing this for every single one of your customers. And it's spinning up agents, managing contexts. It's, it has to do things in a sandbox, so it's also orchestrating sandboxes. Providing all of the context to and from each agent and its sub-agents. And that orchestration is super important. There's no way you can do this without the central orchestration system, and that becomes, in a way, your product.

Tony Holdstock-Brown 7:58 And I think, like, we see this from the leading labs. It's kind of inevitable that this is the way that world moves. And so a guess is what happens with every other product. Like, Datadog is a really classic example. There was this really interesting post on Hacker News this week about agents for monitoring. And why could you not have a long-running agent that also lives in Datadog that takes a look at every SLO you've defined? And it takes a look at every one of your systems and it proactively reports, it proactively suggests different things.

Tony Holdstock-Brown 8:26 That would be sick. And at that point, you've got your personal agents that run on whatever system you use. You've got the products you use that have agents. Maybe they just chat with each other. And the orchestration of these is like incredibly important.

Conor Bronsdon 8:40 I think a backdrop to all of this is how much growth we've seen in the agentic economy this year alone. I saw a stat that in the last year it's been over a 34x in token usage for agents. And that just seems to be accelerating as we roll out personal agents everywhere. Are you expecting the way that companies are working with the Inngest platform to build long-running agents to continue to change the coming months, and where do you see this agentic era going over the next, I know, call it year?

Tony Holdstock-Brown 8:57 Yeah.

Tony Holdstock-Brown 9:12 Yeah. So, yeah, on a personal note, we've seen a big shift away from UI to MCP and APIs, CLI. Um, and that's like the majority of our traffic now. Most of our customers actually don't log into the UI. They use agents to check on whatever it is they've built. Which is insane. And the stats are that people that build with agents are far more likely to activate, like ninety three percent of them activate versus a lower activation rate. And even then our activation rate was super high at like 40% of people.

Tony Holdstock-Brown 9:46 But it's insane. So I think it's number one indicator that that's the way the world is moving. It's much faster, much easier. So I think over time, yeah, like what we'll see is a lot more people building, number one, this way, and number two, um, just taking a look at the way people work with Dots, the way that we've seen our internal work, the way that we've seen our customers work, the way that we've seen people build, it's sort of inevitable that over time you end up sort of abstracting the meaning of tasks and becoming much more higher level.

Tony Holdstock-Brown 10:16 And, uh, a bunch of people have talked about this on different podcasts, like, you almost become a product manager of what you do. Yeah, exactly, exactly, which is great. And it's like so true. Like, I can't see it going back the way it was. Ever. And that means that over time, yeah, kind of expect that people build much more in terms of these agentic systems, and it will only get better. Which is really cool.

Conor Bronsdon 10:23 I've said that.

Conor Bronsdon 10:40 I talked to Laurie Voss recently, who is head of developer relations at Arize, and he's a co-founder of npm, and he calls it product engineering. And I really like that term, where you are, the software engineering has changed. You're still solving problems, but you're not a programmer anymore. You are now engineering the product and the systems around it, like a systems analyst from the 50s and 60s. And to your point earlier about these ideas coming back, it's saying, yeah, we still have to engineer the problems, have to engineer the systems, or problem solving, just the abstraction layer has changed.

Conor Bronsdon 11:11 And I know you have surveyed some engineers and customers around this as well, and a few interesting trends that I saw in some of the data you shared. One is that there are quite a few incidents happening, and people haven't quite solved these systems. I think it was 74% of AI teams that you surveyed had a customer visible incident in the past 90 days. Is concerning. I think where that's people see wave of like cyber incidents and worry, gosh, are we exposing ourselves too much? How do we solve the reliability problem?

Tony Holdstock-Brown 11:41 Yeah, this is, it's really tough, and there's a lot to say about this one. So I'm gonna go from the engineering point of view. Then we're going to go for the tracking point of view. Um, and then I'm gonna go for like a meta point of view afterwards. Engineering point of view is really hard. Um, especially if you talk about a central orchestrator for many personal agents that are long running. Long running by nature means that things are stateful. Um, and it means that if you're doing a central orchestrator for your product, which you must, otherwise it's very hard to for you to orchestrate long-running agents for tens or hundreds or millions of users.

Conor Bronsdon 12:02 Mm-hmm.

Tony Holdstock-Brown 12:12 Um, reliability becomes a big concern. So long running state orchestration typically means that you're going to be building with a lot of asynchronous systems, queues, state, durable execution, a lot of databases that are high throughput to make things work. Observability becomes a pain, and that's sort of our central underpinning of what we tried to tackle in Inngest. Essentially state managed for you, queues, orchestration and so on, just out of the box, and you write a few steps, everything is done. That is tough. And it's even tougher when you consider you're going to need to spin up sandboxes for each of your users.

Tony Holdstock-Brown 12:45 Because if you're, say, for example, Good thing. You've got 100 million users on your platform. Half of them are writing code because of Cursor. And you want to be able to spin on things like Claude Code, that needs to happen in a VM, so then you're orchestrating infrastructure. And infrastructure is like maybe the one immutable piece that doesn't really change because that's always going to exist. Plug something into a DC. Um, so reliability is quite tough with this aspect. And as things scale, I think, like, we haven't quite understood the compounding nature of agents.

Tony Holdstock-Brown 13:16 Yeah. It would be very hard for one user to go through and make thousands of API requests on a daily basis without scripts. You would only have a few users that would go ahead and make those scripts in order to automate their daily life. But now every user is automatically doing that, and every user is going to hammer your API with thousands of requests just to get the context it needs for the agent. So reliability from the infrastructure point becomes a little bit more difficult because the extremes just compound.

Tony Holdstock-Brown 13:44 That's quite challenging. And therefore you need things like flow control, rate limiting, just to make sure that you're good. And agents can deal with that pretty well. If you rate limit your API, it'll just retry. You know, it understands everything generally. So you really need to bake in protections. That's tough. From just to what an agent does point of view, and this is both from a lab point of view and also you building your product, it's very hard to track outcomes. And I think the classic way that people have been doing this for quite some time, and it's not wrong.

Tony Holdstock-Brown 14:11 But it's tough. Is evals, which are unit tests, and a non-deterministic unit test over maybe a subset of all of your data in production. And the non-determinant, non-deterministic user test over a non-comprehensive portion of your user base. Mm-hmm. Very, very tough to understand whether or not your agents are doing the right thing. And that's not to say it's wrong, it's absolutely right, because why would you not do unit tests? But it doesn't track outcomes in production. So one of the principles and things that we've learned from, man, it was Kissmetrics back in the day, who I don't think really do too much nowadays.

Tony Holdstock-Brown 14:48 It's Amplitude, it's all other folks, but like Kissmetrics learned this, and A-B testing learned this. Track events. And then track outcomes based off of what happens. So, for example, if you're at Amazon back in 2004, there's the, does the purchase button being thirty percent larger increased conversion rates? Yeah, probably. Track the events, see what happens, and then track outcomes based off of what's happening. That's like something that we ended up building directly into the platform because we have step.waitForEvent. If you're building a coding platform, and you're doing some sort of software factory.

Tony Holdstock-Brown 15:21 It would be good to track events to see if the review passed, to see if a user merged within some certain amount of time. And if they didn't, you can use these events as a heuristic to say the agent did a good or a bad job, because they're indicative of the quality of the work. And tracking events is super cheap, dude. It's existed for, like, Kafka's so easy. So like you can literally just consume all of the data from how your product works. And then if you're, if you're, if you've got some foresight about what the agent is doing or what the intent is.

Tony Holdstock-Brown 15:52 Track based off the events to see whether or not it did an acceptable job. And that gives you a pretty good deterministic view on how agents are performing. And it doesn't have to be sampled. Like 100% of your agents can be monitored. Which is very different to taking an entire trajectory, throwing it through another expensive LLM as a judge and having another agent guess, did it do a good or a bad job? Which is kind of biased because you're asking an agent that had the same weights to judge itself.

Tony Holdstock-Brown 16:22 So that one's a tough one. And I think that's an interesting point of view, to improve AI reliability and to tell you whether or not, generically, things are performing, is just like we've been doing for products, management in, in year for the, for the past few decades.

Conor Bronsdon 16:40 You've given me so many interesting threads to pull on here. Uh, no, this is, this is fantastic. Making my job very easy. I want to go to one very niche part of what you said, which is statefulness. Obviously one of the challenges that we are orchestrating around with LLMs is that the models themselves are not stateful. You know, they are, as, you know, Wen from Genspark told us, geniuses with goldfish memories. And so we are building the system around them, the agents, the orchestration, the context, the memories.

Conor Bronsdon 17:12 To ensure that they can keep working for us over the long run and actually be more deterministic and solve the things we want to. How do you think about the stateful side of ensuring these long-running agents are more successful over time? And what are the boundaries today of, you know, how long can a long-running agent exist for? What are the ch- context challenges you're having?

Tony Holdstock-Brown 17:38 Yeah, it's, it's super interesting and I think it'll only compound the problem that you stated with, with context and state. As long running agents and personal agents continue to expand. Mm-hmm. Firstly, I just want to quickly note that I, it's really, really, really cool. The LLMs are essentially stateless, and what you put in at any time can just continue as long as you have that state yourself. So that's exciting. I think it's like really tough and I think context management is one of the biggest problems in AI engineering, as everyone knows.

Tony Holdstock-Brown 18:07 It's no surprise. So firstly, if you're having one particular conversation, it's pretty cool if you can do something like durable execution, where every step is recorded, the inputs and the outputs are tracked, and every tool recording, every tool use is recorded. So you can see which tools were generated by the LLM's response, what their execution was, what results they had, and so on. And you get a fully deterministic system. Pretty sick. Because then the state for one function is entirely captured. That means that if you didn't like what happened in one particular conversation, you can replay to a specific point in that conversation.

Tony Holdstock-Brown 18:44 Maybe adjust some of the inputs and see if it would have taken a different path afterwards. Quite fun for testing. But then in the world of personal agents you have this compounding problem of sub-agents and these long running stateful tasks, which I think you've seen with Claude Code for the past few months. Like it writes memories, there's a huge markdown file, files are memory and files are great. And there's like maybe this balance between, do we use a, do we use something like a, like a vector store or do we just save everything in files and have an agent grep for it?

Tony Holdstock-Brown 19:15 Legitimately, I think for us we see people write one parent orchestration agent, which uses essentially step.invoke to run subagents. Sub-agents have the entire context of their own thread, which they summarize and pass back up to the parent agent as you'd expect. And from there, it's really important to be able to take the content from that particular conversation and extract what might be good to save that somewhere, so that you can load that in future. I think like the tough part here is knowing what's relevant.

Tony Holdstock-Brown 19:44 And so this just honestly means that you end up spending more and more tokens to figure out what was relevant and what happened in the past that might be relevant. Then, I dunno, like this is more theoretical now. You get into a problem of like freshness. What was learned a year ago may not now be true, which is really, really, really challenging with. Yeah. Yeah, and that's really challenging to deal with. So like there's still a few unsolved problems. And this is like something that we're taking a look at because it's, it's really, really fun to think about. Um. Mm-hmm.

Conor Bronsdon 19:59 Be, how do you forget memories?

Tony Holdstock-Brown 20:14 My, my personal belief is that like lot of people are taking a look at this from both the front-end lab perspective, a product perspective, internally, like us. Personal agents are incredibly useful. Very cool, extremely scary. Everyone on Hacker News is going to talk about the security implications and I agree with them so very much. But they're still really cool. And so, like, what happens and what happens overall with a long trajectory of personal agents, and what technology does that bring? Think it's gonna be like very quickly unfolding over the next like three to six months, which is like, I can't wait to see it. Yeah.

Conor Bronsdon 20:50 I also think there's a marketing challenge here too, where we are at a state where I think most people in tech are going, oh, yeah, a personal agent makes sense for me, I have these things I want to do. But it hasn't necessarily broadened out. And there is a need to communicate the value of a long-running agent to people's lives. They're starting to see that. You're seeing ads for like, okay, let me help manage your family's calendar, that kind of thing.

Conor Bronsdon 21:15 Uh, but it feels like there is a potential wave here if the broader population starts to unlock and go, oh, I do want that personal agent long term.

Tony Holdstock-Brown 21:23 Yeah. Yeah, yeah, totally. And like guess this is one interesting thing. And then, like, yeah. You've had companies like Apple with Siri take their time to roll out a lot of AI.

Conor Bronsdon 21:35 Take their time is a nice way to put that.

Tony Holdstock-Brown 21:37 I think that's like actually both really good and really bad. But if personal agents get to the point where they're unilateral success, which is possible, and it must be the way that AI progresses, then like it does become just a branding thing and an accessibility and distribution thing. Like, a lot of the times companies are really a distribution game and technology is a lot of distribution. If you've already got distribution, like, for example, WhatsApp or Siri, do we really even need to brand personal agents or is it just like, hey.

Tony Holdstock-Brown 22:09 You can use Siri for everything now. And that's like a general question I don't know the answer to. I think, like, this is where tech just sort of builds on itself. It's, it is very much a branding thing for people that aren't in our circles. Like I can't see my mum going on, like, group-ops and being like, this looks so good. Let me play around with this. It's gonna change my entire life in her 70s, you know? Like, that's not gonna happen. But if she were to use Siri and WhatsApp, like that totally could.

Conor Bronsdon 22:37 And I actually find the hardware push that Meta is doing with Muse really interesting here too, where they're saying we're going to be cute, we're going to give these interesting hardware devices, you can have like a Tamagotchi style Muse experience. I'm very curious to see how that lands with older populations and younger folks. There's obviously a lot of AI pushback outside of the bubble that I think we're in. And so it's going to be a fascinating challenge for companies like Meta to market this to the broader population.

Tony Holdstock-Brown 22:54 Yeah.

Conor Bronsdon 23:08 But there is a massive opportunity, as you point out. I think we're seeing with the huge increase in agentic usage, not just coding, but now like motion graphics, all sorts of use cases, that we are going to continue to just solve things in a way where agents can now do so much more than a year ago, let alone six months ago, is the leaps are massive. And so I have no reason to expect that we won't solve this in a way where everyone starts to get access to an agent.

Tony Holdstock-Brown 23:36 Think the trajectory of AI is such that it keeps getting better. As like the orchestration systems around context and the building of engineering that we do around AI becomes better, and the models get better themselves, what we can do with AI just kind of compounds. And that's really, really cool.

Conor Bronsdon 23:51 I can animate these episodes now that I have no animation background.

Tony Holdstock-Brown 23:55 Insane, like absolutely insane. And engineering becomes much more of a code factory than it does like engineers writing code. I was on a screenshot of my engineer, some of my engineers yesterday and I was writing the interfaces manually. And then asking Claude to kind of help me once I'd got it all done. And they were like, what are you doing? This is insane. Yeah. Like, dude, grow up. Uh, but like as we build better systems. And that is a lot of engineering around the, the, the AI system and context overall.

Conor Bronsdon 24:11 They're like, you're a caveman.

Tony Holdstock-Brown 24:23 It just becomes really great, and then I wonder, like, yeah, there's AI pushback. But once AI is so good that it just does the thing, I don't even think we're gonna be branding it as AI anymore because it just becomes the norm. And I think people's pushback just disappears because the expectations rise. Like you go back to the nineties and you were on dial-up. Okay. You opened up Windows 3.1 and 95 and it was just taking forever. You could hear the hard disk spin up. That was normal.

Tony Holdstock-Brown 24:49 In turning on your computer you would be okay to wait 120 seconds. Did frustrate it. Like if my MacBook doesn't turn on, I never turn it off. But if it didn't turn on in two seconds, I would be so annoyed. Which is crazy, but the collective experience of everyone, expectations are going to change. And people could theoretically push back on AI. But I think like as it gets better, people are just going to get so much more used to it. And I think that's ultimately why I think the engineering behind AI and how you build product is so important, because.

Conor Bronsdon 24:52 I would never do that today. Never. Yeah.

Tony Holdstock-Brown 25:20 If you don't meet that expectation that will be there in a year. A question is, like, it becomes like the whole Salesforce versus CRMs back in the day in the nineties, like, are you building something that people just no longer want to use because they're not going to?

Conor Bronsdon 25:35 I think this is why one of my, my favorite podcasts today is How I AI by Claire Vo, who's a fa- past guest on here. But just explaining, okay, here's how the people I'm talking to are using AI, here's how we can leverage it, here's how I'm leveraging it my day to day. And I think there's a lot of work to be done on that front of just helping encourage people to be ambitious because the models' capabilities are advancing so fast.

Conor Bronsdon 25:58 The orchestration, work from folks like yourselves around enabling those models to be long-running and stateful. Have massive compounding advantages. And we can just simply do more than we could a week ago. Came code a month ago. That went two years ago. I mean, like, this takeoff has really only been like, what, four years now? Ish. And I mean, I look at the like six months ago in AI posts and it's just insane what's happened. Claude Code came out what, last year? Codex was February of this year.

Conor Bronsdon 26:30 Um, and I just don't see a world where, barring a solar flare wiping out all technology, we are going back from this. The tools are just better than they used to be. There are tipping points. We're seeing it with engineers, where engineers who were, I'm not going to use AI, I don't like this, have gone, actually this is really good now. We're seeing it with animation. We're seeing it with people, with marketing. I don't think like writing solve or something. We're seeing it with math, where math has just.

Tony Holdstock-Brown 26:37 The At in Yeah, absolutely Atelena.

Conor Bronsdon 26:58 Completely changed. Think we're going to start to see it with drug discovery and bioscience, though there are still major regulatory hurdles there. And I would caution folks to think carefully about how fast can that actually move. It just seems so clear that the bitter lesson is going to be applied over and over and over again.

Tony Holdstock-Brown 27:12 Everywhere. Everywhere. Everywhere. And that, and that's super interesting, 'cause to bring it back to one of your points, geniuses with absolutely no memory. And so it still means that what you build around AI models themselves is extremely important. And so you might argue, and I've heard this from different people, where's the edge? We're building a company and everyone has access to frontier models, why is my company going to be better than somebody else's company in the same category? And it turns out that, like, yes.

Tony Holdstock-Brown 27:40 You have access to the same models. But the way that you manipulate that model, the way that you store context, the way that you guide it, could be very different in the experience you create, even if it is a set of APIs, can still be fundamentally different. So I think, like, the engineering around AI still becomes the most important thing, even if you're using Claude Code to do that. Like, it doesn't change the fact that you still have a bunch to differentiate on.

Conor Bronsdon 28:07 Let's get into some of the nitty-gritty of those decisions. I've seen you argue for explicit step APIs over compiler directives like Vercel's. How are you thinking about some of the specific infrastructure decisions around APIs and other aspects?

Tony Holdstock-Brown 28:23 Yeah. Um, I'm gonna go back to our principles. And then I'm going to talk about the future of, uh, how things might work. Especially with compounding AI. Our principles when we started the company were meet you where you're at. Um. Mm-hmm. Preface this by saying I very much like Cloudflare. It's the opposite of Cloudflare. Cloudflare, like, we've built some cool systems, you're going to have to re-architect to build on Workers, and that will solve all your problems, which is great. Workers' platform is pretty sick. We were very different in that you could drop your SDK in Python, Go, TypeScript, run it on Lambda, run it on GCP, run it on EC2, it doesn't matter.

Conor Bronsdon 28:36 Mm-hmm.

Tony Holdstock-Brown 28:59 And it would just work. So meet you where you're at, be very flexible and have it do the right thing based off of an SDK. And that's quite cool. Because then it does just work anywhere. So our principles are, if it's simple, and Amisha, where you're at, then generally speaking, it's a better experience for both humans and agents. And that's super important going forward. Secondly, it shouldn't do fancy stuff and it shouldn't be surprising. Because the more surprising it is, the more thought both you and your agents have to put in to understand things.

Tony Holdstock-Brown 29:28 Which might be short circuited with a lot of the training that happens with models, but it's easier to be simple. Um, for us, I think like the step API is good. Because it gives you a clear definition between transactional boundaries in your code. That's great. But it's also important because, for example, one of the important things in our, in our system is step.waitForEvent. And that's actually used under the hood a ton. For example, if you've got one parent agent, and it orchestrates four sub-agents.

Tony Holdstock-Brown 29:59 Each of the sub sub-agents need to tell the parent agent when it's finished. And in order to do that there needs to be some mechanism. The typical way of doing this every time you are building a system was to store an entry in a database like a signal, pass that signal down to the sub-agents and say, tell me when I'm done. Yeah. Terrible from an engineering point of view because it's just direct coupling that sucks. It's very hard to change. step.waitForEvent inverse the model. Where an agent just says, yeah, I'm done.

Tony Holdstock-Brown 30:27 And you set up a signal to say anytime an agent is done, if it matches this particular pattern, if it's like this agent ID, let me know and I'll automatically resume. That's really cool because if you're doing classic sort of RPC thing, you are doing an e-commerce payment, you don't need to know the 10 downstream workflows that need to run when that payment finishes. You just need to say this payment was done. And so the primitives around how we built our thing are very, very different to say, like, put it in a compiler.

Conor Bronsdon 30:35 Mm-hmm. That's.

Tony Holdstock-Brown 30:56 The primitives carry over from any language on any system. And it means that if you rewrite something from TypeScript to Python because you decide you want to be more AI forward, your functions will pick up where they left off because it says it's the same step IDs. You switch from AWS to GCP, you do a cloud migration across regions. Cool, it'll pick up where it left off. So less surprising, maybe a little bit more powerful. But less magic. And the magic is cool. Yeah. I dunno, when you've been doing this for so long I don't really love magic anymore, you know?

Conor Bronsdon 31:27 What about the human element? So you mentioned trying to build simplicity for both agents and humans. And I think that's smart given that agents are increasingly the drivers of many apps, the people who are using many of our tools. How do we ensure that human judgment is being ingested and pointed at the right directions?

Tony Holdstock-Brown 31:50 Tough. Overall. There's a few different levels to this, the engineering level, the architecture level, and the overall direction. An engineering level, I think like some people care less and less. We care more and more, some people care less and less and that's totally fine. Mm-hmm. Th- I think there's a view if you're, especially if you're building something for yourself. If it does the task, does it matter? For us, if we're an infrastructure company, well, yes. But if you're building your own personal project, maybe no, that's arguable, but maybe no.

Tony Holdstock-Brown 32:18 Um, from an engineering point of view though, the simpler, the better, because it's just easy to understand. Like you're saying, general software architecture principles. Yeah. The stuff from Clean Code and whatnot back in the day. Good book. I think it actually got some flack over the past four years. Um, but still interesting book back in the day, that stuff is still relevant. From a directional point of view, my biggest problem with agents are that agents will nitpick on everything and give you an infinite amount of work to do.

Tony Holdstock-Brown 32:48 And this is the toughest part. You go from something like a Greptile or a code reviewing bot, nine times out of ten you're getting a four out of five. And the four out of five is because of some crazy edge case that absolutely won't hit. And it's maybe because you were writing something, you had a database migration, you adjusted the database migration in that same PR, and then it was like, this is going to fail because once you deploy this, all of the previous stuff is going to change. I'm like, you haven't deployed this.

Conor Bronsdon 33:12 Let me push back on this a little bit. We are seeing that AI is so much better at chaining vulnerabilities than ever before. Should we not be much more careful about these bugs?

Tony Holdstock-Brown 33:24 I agree. I totally agree. And I think maybe the framing of like the nitpicks, the edge cases that would never happen was maybe bad. I think it's like, if you would like to improve this API, here are eight next steps you can do. Rather than like, here's an edge case. Yeah, exactly. So, bad about that. Yeah, I totally agree with you. It's just, I think, like, yeah, sure, handle the edge cases, make sure that things are good, handle idempotency, make sure there are no conflicts, you know, do things transactionally.

Conor Bronsdon 33:37 You have to do this. It needs to be, here's our.

Tony Holdstock-Brown 33:47 All the standard stuff applies with software engineering. Once you've built an API, it is like, here are eight things that you could follow up with. You have to have an endpoint, and agents are really good at just layering on more and more. I think, like, you take a look at, I don't know if you've, you al- use like chat or Gemini on your phone. Every time you ask a question, it will follow up with like, would you like to learn more about that? It's like their training where they're like, oh, here are eight more things you can do.

Conor Bronsdon 33:54 You have to have an end point.

Tony Holdstock-Brown 34:11 And to answer your first question about like the specifics about like judgment and so on, I think we need to be really, really good at figuring out the intent of what we're asking. And when we're done and the direction. That, uh, I think that taste, intuition, um, intuition is a really big thing in our company. The intuition of what to do is extremely important. Yeah. I don't think that'll ever go.

Conor Bronsdon 34:32 Know. I tend to agree. We are at a point where agents can come up with good ideas based off of the corpus of information around them, but human expertise is so important to drive that and to find the areas where they need to stop. Oh.

Tony Holdstock-Brown 34:49 Yeah, yeah, yeah, yeah, yeah, yeah, yeah, definitely. Then on your note about security, like did this week was. Crazy. Like crazy. If we're talking about long running agents, I think it's absolutely critical that we do this from a security point of view as well. You know, like one of the easiest wins for people would just have, would be to have a long running agent that supersedes Dependabot. Please. Supersede Dependabot for the love of all that's holy, get rid of Dependabot and make it better. And have something that just continues to improve the security posture of your app.

Conor Bronsdon 35:13 Or, uh, though.

Tony Holdstock-Brown 35:23 Because the edge cases that you mentioned are like absolutely critical. And if we don't do that, I think like the discourse around Hacker News and everything on everyone on Twitter is like, it's much easier to be red team than blue team. Obviously, entropy and whatnot agree. Yeah, hope. Get help with agents. Like, it's, it's good and bad all in one because maybe they'll be misguided. Mm-hmm. Honestly, if red teaming with an agent, it finds stuff, you can obviously red team as a blue team. And then and help. Well, th- that is just the, the blue team. You don't need victim, yeah.

Conor Bronsdon 35:51 I, I've been experimenting with this just on my personal code base, which is not that production bearing, beyond this show. But having, in my case, I'm having like a DeepSeek v 4.1 flash agent that is just continually checking every day, like, where are there potential vulnerabilities, and then passing off issues based off of that to coding agents to go solve. And I think we're seeing that happen at scale at some companies. Anthropic released a great article recently about their code review process.

Conor Bronsdon 36:19 And they talk about turning bugs into evals, and they talk about turning on these security loops. And I absolutely think every company needs to be thinking about what is my security loop, because we are going to continue to have so much more code, and we have to ensure that we are beating attackers to the punch on it.

Tony Holdstock-Brown 36:34 Yeah, yeah, exactly, exactly. I mean, to bring up to the point of building long-running agents, it becomes once again the, the challenge between, some event happened. Code was pushed. We need to orchestrate a bunch of agents around that code, similar to code review, just to make sure that things are good. And this needs to be a continual ongoing process. Like the memory that you talked about. What is valid now, and goes in now, may turn out to have a bunch of security issues if you have a dependency somewhere in your supply chain.

Tony Holdstock-Brown 37:05 And the long running agents come into play because you need to make sure that what was good now might not be good in three months' time. And that needs to be fixed, even if it's not relevant to code that was being pushed today.

Conor Bronsdon 37:15 One of my favorite loops I've seen recently was, I think Rhys Sullivan, um, on X posted about it. It was, hey, I want you to look at my app every day and figure out how to make it faster. I just want to have everyday loop of like, how can my app be faster? How can it be better for both agents and humans? And I think there's so many opportunities that we just need to have a long-running agent that is checking regularly and say, okay, this is my one job. I'm going to keep solving this. And then orchestration around that.

Tony Holdstock-Brown 37:41 Yeah, exactly that. And I think, like, to, to talk about the principles of how we'll build our stuff and steps one more time versus directives and so on. A step is really just souped up tracing. To be frank. Tracing is a step in sort of open telem- telemetry that stores the output, then it reruns your function if it crashes, and each of the traces just re-injects the state that was saved previously. That's an easy way to think of it, memorization. But because you're breaking things down into steps, you get observability across every single facet of your function.

Tony Holdstock-Brown 38:14 And if things were taking a long time and one step was slow. Sick. Totally easy to find out. You can use insights to query for slow steps. And then you can use an AI agent to go through and then improve the perf of your functions. And it really comes down to, as everyone knows with AI, the data in, data out, just how much data are you getting from the things that you've built? And how easy is it to continually improve? And like to that effect, like I would love to add.

Tony Holdstock-Brown 38:40 We have this thing called derivative endpoints. Essentially workflows, but in an API endpoint, and if your API fails, it redirects you to another URL that blocks and waits for the retry. We go and retry your API endpoint. Pick up where it left off, finish the API result, grab that, send that to user on the redirected URL. But it would be sick if you could just add SLOs to your API endpoint. And if it didn't hit SLOs, triggered something. Goes through, looks at the stack of traces that didn't meet the SLOs, investigated the reason.

Tony Holdstock-Brown 39:09 And then automatically push fixes to your entire code base to make it faster. Like, that is stuff that's totally possible. No human really wants to do that. Like, no human loves the idea of. And there are really good SRE teams that are really good at this. No rare. People logging onto dashboards every day, looking at an SLO and then being like, I need to make this faster, and then going on an eight-hour investigation to find the root cause is an entire job. That is draining. But an agent can just do this. Like that entire thing around continual improvement is like totally possible.

Conor Bronsdon 39:35 Mm-hmm. Mm-hmm.

Conor Bronsdon 39:44 And I think for anyone who is interested in problem solving today, there is so much that you can do, and this is where I would encourage our listeners, be more ambitious with what you try to achieve with agents. You can do more today than you ever can. I want to come back to a point you made very early on in this conversation about using product signals to identify the success of what you're building. And to me, this just speaks to the value alignment there, if you can build great loops, you will get these product signals as long as you can trace what's happening.

Conor Bronsdon 40:17 Run some evals, maybe not all of them in a production. I think you did a survey where you found what, 35% of teams building AI in production weren't actually using evals at all.

Tony Holdstock-Brown 40:27 Just, just YOLOing it. Which is okay. Models are pretty good nowadays. Like, that's, that's, that's a reasonable thing to get something out the door. But you wanna make sure that it's meeting the need of your customers.

Conor Bronsdon 40:28 And. Yeah.

Conor Bronsdon 40:38 Speaking of customers, I want to ask about platform scale. I think something that people run into is the constraints of compute and the constraints of the systems you've built. How are you handling customer queuing for these long-running tasks as your platform has scaled out so rapidly?

Tony Holdstock-Brown 40:55 Yeah, or can I tell you a secret? Well, we ended up running out of compute.

Conor Bronsdon 40:57 Please. I'm not surprised. Yes.

Tony Holdstock-Brown 41:01 Yeah, yeah, yeah, yeah. We run on multiple different clouds and we also have bare metal. And we ended up acquiring a bunch of new bare metal machines and then racking them ourselves with, with switches because compute is just crazy hard. Bandwidth in our game is really, really tough because we're doing a lot of model calls for our customers. We're storing a lot of state and it's just, public clouds have got this like hold on you with bandwidth that is very hard to, to break through. So we did a bunch of stuff bare metal and that's great.

Tony Holdstock-Brown 41:27 Um, we ended up building, uh, a mi- a multi-tenant system that works in cells essentially. And that's really, really cool. Um, because, yeah, basically makes it cheaper for everyone. Instead of spinning up like an NVMe disk per customer and then storing a bunch of state. Because state is essentially the key value store for each particular step that runs. A function run finishes, that state gets deleted. Super high throughput. Super high read, super high write, super high delete. Worst case for all database systems. Rinses NVMes, go through the entire drive-write per day, is not fun.

Tony Holdstock-Brown 42:00 Doing that on a per customer basis and single tenant is, is hard possible. Yeah. Easier for everyone involved, but it comes with risks because you have multi-tenant architecture, a lot of issues around that. We've been investing a lot in cell-based architecture. Queuing. We actually built on FoundationDB when I started the company and then switched to, uh, to, to something Redis compatible. Which, in my opinion, was awful. But it was the right decision because we wanted people to be able to run it open source. And FoundationDB isn't something that people want to run open source just to develop locally.

Tony Holdstock-Brown 42:35 So, our entire view of meeting you where you're at is that you can run Inngest in one command and bring up the entire production-like infrastructure, one command, and then have everything run locally in exactly the same way it would when you push it to the cloud. In order to do so, you really need not just a fake copy of FoundationDB, but the actual systems running on your service, on your machine. Hence open source compatibility with everything. Even if FoundationDB is open source, having a prerequisite to install that wasn't gonna work.

Tony Holdstock-Brown 43:04 The cloud does work a little differently now behind interfaces. Um, there's a bunch of different changes. FoundationDB is back because scale. It's, it's quite a challenge. Uh, it's quite challenge, but very fun.

Conor Bronsdon 43:15 Are there any particular learnings that you've had as you've gone through the scaling process that you would want to share with the audience?

Tony Holdstock-Brown 43:22 Yeah. And actually, the, the folks at PlanetScale talked about this with Neki, um, just last week.

Conor Bronsdon 43:30 PlanetScale is just so good.

Tony Holdstock-Brown 43:31 Yeah, yeah. They talked about something with Neki and the things that they do with SQL parsing in exactly the same way we talk about it with events. Turns out that not doing work is the best if you're trying to hit scale. Don't do needless work. So for us, let's take step.waitForEvent, which is so good as a user and so awful for us to run as a company. Um, let's say you run a million functions. Which is not that many. And you do step.waitForEvent.

Tony Holdstock-Brown 44:02 You're waiting for a million unique events. Maybe a million unique users are doing something and each of those is a different expression. Any time you send one event to us, we need to check all million events to see if it worked. And you've got a million wait for events, you could assume that we're going to get a million events. That's a million times a million, which means we're now doing trillions of quadratic expression matchings, which has to happen in real time. Otherwise you get huge delays on your function reson- run resuming from step.waitForEvent, which is not what we want.

Tony Holdstock-Brown 44:31 So the answer really is to break down the expression into a bunch of inverted indexes, see the expressions that come in, see which might match, filter out all of the expressions that might not be valid before you even evaluate them. And then you can go to evaluating in what would take hours in a millisecond. And it turns out that exactly the same as PlanetScale, don't do the work you don't need to. And then the standard engineering, you know, SLOs, take care of things that you really need to take care of from a memory point of view.

Tony Holdstock-Brown 45:00 Make sure that you're not doing extraneous CPU processing, which is the same principle, but. Um, scale is, uh, scale is tough. The other thing I suppose would be like, even if you want to be extremely generous to your users, which we do, you still need limits in everything. AWS has limits in everything. You need to ask them to raise limits on Lambda. You ask them to raise limits on SES. Just, everything should have limits lower than you expect and you raise them over time once you're, once you're safe.

Tony Holdstock-Brown 45:26 And I think that this is even more important, bringing it back to AI, because agents are going to do things ten times more than your customers would have two years ago.

Conor Bronsdon 45:36 Yes. We simply have to scale more than we used to. And Tony, I really appreciate you taking me through the approach that Inngest has had to this scaling. Fascinating to hear about some of the perspectives and approaches you're using. As we near the end of this conversation, what are the things that you would say to engineers, founders and leaders listening about the best practices they should take forward as they also look to scale for the agentic era?

Tony Holdstock-Brown 46:05 Yeah. I think like this goes back to the conversation and theme of long running agents. The long running personal agents are relatively new, but it becomes interesting and apparent that this is the way that people are going to consume other products. So thinking about how your product works, not just with an agentic experience, like how do agents access it. But literally, how does it work if your user never sees your product overall? That's incredibly important. How you orchestrate your own agents to solve your users' problems, long running as they may be, is super important.

Tony Holdstock-Brown 46:39 The infrastructure around that is incredibly difficult because not only are you building your original product, you're now building a system of long-running agents around your product to empower your users. And the choices you make around that are super important and really, really tough.

Conor Bronsdon 46:54 Let me throw maybe one more thing at you here really quick. If I was trying to build my first long running agent, and I said, I don't want it to be a personal agent through Muse or Instinct or whatever, I want to build my own. How should I get started?

Tony Holdstock-Brown 46:55 Cool. Yeah, yeah, yeah.

Tony Holdstock-Brown 47:07 Mm-hmm. Yeah. This is a, this is like how to b- how to draw an owl. You know, draw the circle, draw the rest of the owl, dude. It's really tough. Um, and I guess like it depends on the overall objective. I think, like, firstly, it's pretty clear that you can't have one agent do everything and that sub agents are pretty good. So figuring out some architecture where you have a request going. Creates an agent. And that agent can delegate to other sub agents as necessary is super good.

Tony Holdstock-Brown 47:35 And that's really good because then you get almost unlimited tries because each sub ad- sub agent has its own context that can go completely haywire, report back to the parent agent and it doesn't derail what's trying to happen. So architecture around one input, series of steps, sub agent delegation, particularly important. It depends on the experience that you're providing to the user. Do you need real-time UI updates? Do you need chat? Do you need steering? At which point then you need to interject some state into those functions.

Tony Holdstock-Brown 48:06 Which is a choice for you to make overall about the product experience. Um, overall, you need to take a look at queues. You need to take a look at workflows and you need to take a look at state, so that you can track what your agents are doing and come back to it to see if it were doing the right thing. Yeah. Tons of different frameworks out there for building with agents, there are tons of different LLM providers. Honestly, at this point, I think the agents at LLM and models are so good that you can just.

Tony Holdstock-Brown 48:34 Straight up use an OpenAI or Anthropic request and get everything that you need nowadays. Um, but the tracking around, uh, around what you're building and, and how the agents work is the most important thing. I think secondly, tracking the outcome is so important. I'll sort of take a hypothetical point of view, which is talked about security, we talked about like the Datadog view of like, make my app faster. If you had a long-running agent that you wanted on your infrastructure to consume something from, say, like, Datadog or like some other APM.

Tony Holdstock-Brown 49:06 You would then have a goal of a PR that wanted to be merged. And then you would have the agent pause after the PR was created. You would have the agent resume once the code was merged, tr- to check whether or not that PR had an impact on said SLO. So you're going to be storing a lot of state. You're going to be trying to revive this state once the PR is merged, which means a lot of events and event-driven systems to make sure that your agent is revived in the right way at the right time.

Tony Holdstock-Brown 49:33 Orchestration is like, the orchestration is the most important thing there. Legitimately. Still early on the long-running agents, but like when you have that talk in the future, I can't wait to watch it. This is, I think, the biggest directional shift in AI that we've had since, like, maybe tools. I can't wait for it.

Conor Bronsdon 49:50 Yeah. I think it's going to be fascinating. And I know we're going to be writing about long-running agents on our newsletter at newsletter.chainofthought.show. And so I'm very excited to get more thoughts from you and more feedback, because I think there's a huge opportunity to explore this topic more with our audience.

Tony Holdstock-Brown 50:07 I just want people to, I want to see people's use cases, man. I want to see people's thoughts. Like, I want to see how it happens. It would be so interesting for you to demo if you had personal agents doing things. Like, just how it works. Duet is going to be good. We were trying, some, some folks in our company were trying, uh, trying an OpenAI's version, but they were like, you can only run one agent at a time. No handoff, relatively weak. Then you go to Dots, like you go to Grokbots, and it's like an entirely different game.

Conor Bronsdon 50:34 I've liked Grokbots a lot and I, I'm really impressed by how much they're scaling it out. I need to spend more time with that. That's.

Tony Holdstock-Brown 50:38 Crazy. Yeah, it's crazy. So, I think like one of our goals is like to, to make it easy for people to build personal agents overall. Inside their product in a similar way.

Conor Bronsdon 50:47 And the data advantage you get from something like having access to all that data of X Grok bot is, is so huge too. So.

Tony Holdstock-Brown 50:52 Yeah, yeah, exactly, exactly. And like specifically, honestly, outcome tracking. Like we released it and then a bunch of people just started naturally using it and we've not really talked about it, which is fine. Um, but the outcome tracking of a hundred percent of your agent trajectory is like great. 'Cause one of our long-term views is you can take all of that data. And if it's known good, because it had a good outcome, you can use that to help improve your DRAI, whether or not that's fine-tuning or open weights models. Like it doesn't matter.

Conor Bronsdon 51:18 How can folks use that with Inngest?

Tony Holdstock-Brown 51:20 So, um, the, the core premise right now is, you write your entire AI agent. And then you score it based off of events. So for example, you're doing a coding agent or a code review agent. If people thumbs up, thumbs down, if they resolved the comment without pushing a commit, then you know it wasn't really a good comment. So you take all of those events, you take a look at the state of said events and what happened around them, and then you get your, you get like a good score.

Tony Holdstock-Brown 51:45 Yes, no, good, bad. You can give yourself like a score between zero and one if it's not quite Boolean. Um, and that results in every single trajectory having a group of scores that tells you how good the outcome was. And you literally just group them. You just search for all of your runs and you group them into data sets as good or bad. You take all the bad ones and you improve them in a very similar way to evals. You take the good ones, you download the trajectories.

Tony Holdstock-Brown 52:11 Sorry through something like Tinker. Fine tune your own open weights model. Throw it up. We have step.experiment. Which is nothing novel. It's A B testing, dude, Optimizely. Dude, it's this exactly. It's like, people would be like, it's the most world-changing thing. You can A-B-test models. No, dude, it's like the same stuff we've been doing in 2013. It's 13 years old. Um, you can like A B test whatever you want inside your workflows and then see if it did the right thing. So like I, I'm really excited about the future of that. Yeah.

Conor Bronsdon 52:22 We're all doing A all the time. Yeah.

Conor Bronsdon 52:32 But.

Conor Bronsdon 52:42 Amazing. And I'll say I highly encourage everyone listening, if you have found this interesting, check out the work being done at inngest.com. There's a ton there from their team and a lot of learnings from Tony and everyone else, uh, at Inngest. And if you enjoyed this conversation, and you've sat with us through it, thank you, first of all. But please just drop a quick comment. Let us know what you enjoyed. Let us know what questions we didn't get to you that you'd want to hear from Tony next time. And thank you so much for having us down to your offices here in San Francisco. It's been a blast.

Tony Holdstock-Brown 53:10 Yeah. Thank you so much for having us. It's really, really great to chat with you. Super fun. Yeah.

Conor Bronsdon 53:13 So fun. Thank you all for listening and we'll see you next week.