The economics of agent scale: tokens, ROI, and building platforms for AI-first teams (Part 2)

In this episode, Stack Overflow Director of Platform Engineering Peter O'Connor continues his conversation with Andi Gutmans, this time digging into what changes when you're running dozens or hundreds of agents instead of a handful of humans clicking buttons. Andi explains why he thinks the model itself is rarely the bottleneck anymore. In his view, the harder problem is figuring out the least amount of context needed to get a reliable outcome at the lowest cost, and why "token maxing" is the wrong goal entirely. He also shares how a weekend prototype changed the calculus on de-risking new ideas, and lays out what platform engineering teams need to build to treat "the agent" as a persona in its own right. The discussion also: - Breaks down why cost governance matters more at agent scale, where a single employee might be responsible for dozens of agents working around the clock. - Covers Google's approach to "hill climbing" model and data together, and why Andi is skeptical of vendors claiming to have solved the context problem. - Explores what new tooling and observability platform teams need to build to serve "the agent" as a persona alongside developers, data scientists, and business users. - Closes with Andi's advice for someone starting a computer science degree today: fundamentals still matter, agentic fluency is now essential, and business judgment may matter as much as either. Notes TRANSCRIPT Eira May: Hi, and welcome back to Leaders of Code. This is a segment on the Stack Overflow Podcast where we get senior engineering leaders together to talk about the work they're doing, how they build great teams, and the biggest challenges they face in the market right now. My name is Eira May. I'm the B2B editor at Stack Overflow, and this is the second half of our conversation with Peter O'Connor, Director of Platform Engineering at Stack Overflow, and Andi Gutmans, who is Head of Agentic Data Cloud at Google. I hope you really enjoy the second half of their chat. Peter O'Connor: You discussed about token efficiency, and that's becoming a very, very big thing lately. Where do you think... I agree, I don't think models need to get much better. I think people still opine about Opus 4.6, about how amazing it is. So where do you think models have to go from here? Is it really an improvement? Do you think it's now becoming more in-house? Are we always going to be using frontier models? Where's the next step you think we're going to see you taking in the model industry? Andi Gutmans: Look, I think models will definitely keep on getting better, and I think we're seeing that every model that's coming out of DeepMind, models that are coming out of other players, right, they are getting consistently better. So I'm not saying that they shouldn't get better and they won't get better. But what I am saying is that I think for a lot of tasks that we're trying to automate today, a lot of the models we have today are actually good enough. Now, I think about this problem more as a system as a whole problem. It's not just about the model, right? It's the model, it's the harness, it's the context, it's the personalization. And the way I think about this is you want the least sophisticated model that gets the job done at the lowest cost. So it's very use case specific, right? In some use cases, it may be a super lightweight model. I mean like Gemini 3.5 Flash is awesome and it's super low cost, right? But you're going to have situations where you're going to want to use Gemini Pro. And so I think we're going to get better at figuring out what model to use when, how to make sure that we are limiting the context to only the context that matters. So we actually need to do less token processing, plus the models have more accurate trajectories. So you're going to have less of a reasoning loop, which is also going to reduce the amount of tokens. So I think a lot of the prompts shift towards that system as a whole problem. That's where I think Google is very fortunate because we're actually the only hyperscaler who has both in-house model development and has a differentiated data platform and understands information really well. So that gives us the opportunity to hill climb in partnership with DeepMind model and data platform together, really thinking about this problem of the least amount of context that drives outcomes you need at the lowest cost. Peter O'Connor: Yeah, that's super valuable. I think I'm with you on trying to figure out how data plays in. The systemization's a really smart thought experiment when we think about how this works. I was talking to a friend the other day and saying the one worry I have recently is when I see these harnesses and people are talking about agents just getting better and better in these models. I keep thinking about, for lack of a better word, but drills. I want to make sure I get a model. That system of the drill system to me matters. I don't need you just to make a better drill that makes a bit that drills a hole better, but now drills can paddle mix, they can clamp things together, they can cut wood. If they get too specialized, that's too much of a problem. So I really like that systemization point of view to it, so. Andi Gutmans: Yeah. Peter O'Connor: When you think about data and you think about the skills and the context, you're talking about the least amount of context viable to make the job done, which is great. How do you measure that? How do you approach a problem space like that where it's like, yeah, here's the scope of knowledge you need. Why? Because this is all I've ever known, but really all you need is this. What's your thinking behind scoping that problem a little bit better? Andi Gutmans: So it's actually a really hard problem. So that's why I get a bit suspicious when you have players that say, "Well, I've got this catalog and I'm enriching all this data. It's going to make models better." You actually have to hill climb model and knowledge together and actually have the right level of evals to really understand what part of the context is actually moving the needle. Because you're optimizing for agents, you're not optimizing for humans. And I think what I see is a lot of companies are approaching this in the old-fashioned way, which is like, oh, I'm going to let humans build the curated ontology and that's going to make my agents better. But the reality is that's theory, it's not practice. So we're going after this in, I would say, much more of an AI first thinking way, which is our customer is the agent. How do we make sure the agent gets the right outcome? And it's just a ton of work on the eval side, really understanding the trajectories, working with model builders like DeepMinds to make sure that if anything has to change in the model, it changes in the model. Otherwise, it changes in our context. And we're bringing all of Google's differentiation to the table there, both on enrichment, on search. Search is a big deal, which is how do you know which context actually matters? Most vendors don't talk about search. I put this as probably at least in the top three problems of solving. So it's a big difficult problem. Any vendor that will say they've solved it fully is not being accurate. We haven't solved it fully, but I do think we are on the right path to making sure we can really optimize these dimensions. Peter O'Connor: Really warms my heart to know that you haven't solved it because internally we have our own B2B SaaS product we're going out with, Stack Internal. And our position is like, hey, the context problem is difficult. What we want to try to do is have agents solve a lot of your issues and inject a human in the loop as necessary, right? Because we know they can be the authority and that'll help you achieve your business outcomes today, so. Andi Gutmans: Exactly. Peter O'Connor: I'm so glad to hear that. Andi Gutmans: Yeah. And by the way, we have seen that human intuition on what is helpful does not always work, which is why you need a more systematic way to tease that out. Peter O'Connor: Yep, for sure. Humans tend to believe things that aren't really true sometimes, but we're fallible. Andi Gutmans: Exactly. Peter O'Connor: Oh, gosh. Andi Gutmans: We also hallucinate sometimes. Peter O'Connor: Yeah, we do. We do some real amazing thing. If tokens are becoming expensive and we want to make these context windows just right and we have to pull in all this data and we have to do agent ontology creation and mapping, how do we generate a system of understanding, like return on investment? How much do I need to invest in a data platform versus making sure my harnesses are right, making sure I made the skills right? Is that in your brain space at all? Are you thinking about that? Andi Gutmans: Yeah. I mean, not to sound too self-serving here, but it probably will sound a bit self-serving. I think that's a bit of the benefit of we have the whole stack. We've got the GPUs, we've got the models, we've got the data platform. And so our goal is to actually make sure this is all integrated and optimized so customers actually have a pretty easy time getting to the outcomes they need. We're also open, so we'll also work with third party models and so on. But the reality is for most enterprises, solving this on their own is going to be too hard. They do need to work with folks. And I think we're pretty good at it as Google. It's working with folks who can bring that all stack together and then also make sure that that stack is super cost-effective. So for us, it's very clear that it's not only getting to the best outcomes, but we also have to be the most cost-effective in the industry, right? And so we're also consistently working on driving down the cost of being successful with AI. And I think our customers are feeling that. Peter O'Connor: Yeah. Yeah. You've mentioned cost a lot here. Is cost, is that up and down the market segments? Have you heard? Is it from Epic enter