No Dumb Questions: What is AI context architecture? Why not just build your own?

In this No Dumb Questions, Phoebe asks Stack’s Engineering Manager Doug Whitley and Product Manager Ash Zade everything she wants to know about AI context architecture. What exactly is it? Why is it so important? What makes for good AI context architecture? Why buy one when you can build your own?

What is AI context architecture?

Phoebe Sajor: Hi Doug, Hi Ash, thank you for joining me for this No Dumb Questions. To start-what exactly is context architecture in AI and why do we care?

Doug Whitley: So, it's kind of the same thing that regular context architecture is when you're talking about the actual architecture in buildings. AI context architecture is about fitting everything around the AI into the system-what you're viewing, working with, and seeing. It's easy to get caught up in minute details when it comes to AI. You may want to have a very focused context that you're keeping in your system. For example, with AI I don't necessarily need to know the thousands of logs that went into the work I’m doing, but I do need to know the information in a very specific set of logs when I'm trying to solve a problem related to them.

Ash Zade: As a Product Manager, I think of AI context architecture as the guardrails and constraints that we can provide for AI agents. What you're trying to do is remove the ambiguous decision-making and variables from AI agents so the user can have more predictable outcomes. It might look like setting up a system that tells your AI agent, “Only look at this data and given that data only do these three things. If you get stuck, do these two things.” So, not only does the agent have the limited context it needs to work off of, but also the context of what to do when it runs into new situations.

Context infrastructure vs. context architecture vs. context engineering

Phoebe Sajor: I recently interviewed Michael Foree on this show and he taught me a little bit about context engineering. How is context architecture different from context infrastructure? How is context architecture different from context engineering? How do they all work together?

Doug Whitley: The best way to understand it is to start with context infrastructure, which is when we start getting into ways you're actually serving context. Infrastructure is how you deliver something-like context-or how you organize what you're going to deliver. So for context infrastructure, you'll see things like context libraries and RAG-specific context infrastructure. It’s the mode of storing, surfacing, and giving context to an AI agent so it can solve a problem for you.

When we talk about architecture vs. engineering-regardless of the word context in front of it-the lines get a little blurrier. They really go hand-in-hand, but architecture is a little more philosophical. Architecture asks why we might try to do things in certain ways, why we might try to put things together to achieve certain goals. Architecture is the design. The standard design patterns engineers use, for instance, is architecture. Engineering is really where the rubber hits the road-where you're actually building stuff. You might decide to use this particular algorithm because it's very effective or you might use certain languages because you’re building something with RAG.

Where RAG and MCP fit

Phoebe Sajor: So where does RAG and MCP fit into the AI infrastructure vs. architecture vs. engineering discussion?

Doug Whitley: When you're talking about MCP, that's architecture. MCP is a defined protocol that has requirements, but how you implement it-how you design it into the system-is completely up to you. You can implement MCP in any language, or you might choose to only implement certain MCP server features or only support certain MCP clients. You’re making decisions about the design of the AI system.

RAG is a little bit of all three working together. You're going to have some sort of infrastructure in the back of the RAG-it might be your indexing, it might be a context store that you've done highly performant query setups with, or something else that delivers context and makes it searchable. Maybe it’s a list of your rules and processes for a coding agent saved in Markdown files. The way you store and serve that context to your ML or LLM model is going to be the infrastructure.

But RAG is also engineering. For RAG, we might build it in .NET, we might build it in Python, and if we're feeling really spicy, we'll build it in Rust. We can build it many different ways, but when we do, we have to follow an architecture. Think of it like this: if we're building a car, the fact that we're building a car in the first place is an architectural decision. We decided we wanted a car, not a boat, not a scooter, not a bicycle. The make and model are the engineering side of it. Where we buy the parts is the infrastructure.

How context architecture makes AI agents work better

Phoebe Sajor: Ash, from a PM product perspective, how exactly does context architecture make AI agents work better?

Ash Zade: I'll borrow Doug's car analogy. We can ask an agent to go out and learn everything it can about different tire options and come back and provide recommendations. If that agent walks into a library and looks for tires, it’ll find information on tires for planes, bicycles, and wheelbarrows. It’ll find matches for tires anywhere because they're all tires. It's valid. But if we’re building a car, we only care about car tires. But no matter how much you try to set up a guardrail for the agent to only look up car tires, if you’re doing it in a prompt you can't guarantee it's only going to come back with car tires.

What you want to do is set limits on the information you give the agent access to. When you control what it has access to, it's not going to drift and bring back information about bicycles. Here's the knowledge I want you to care about and use in the task I'm giving you.

Another part of building context is agentic memory-making sure the agent remembers the work it’s already done. It’s checked for sports car tires, van tires, and truck tires. You can refine it and instruct the agent to only focus on sports cars. But you can’t build a sports car in a day, so you’ll want the agent to maintain that history and remember it’s working on a sports car. Then you can go away for a day and come back to keep working on the sports car, even when there are multiple things your agent is working on. Part of AI context is saving what the agent is doing, what it's learned, what it's communicated, and what it's built so far. If you multiply this with 10 agents or 100 agents, it’s especially important for you to maintain all that information so you can build upon it.

So you have retrieval and the maintaining of context. But you also want to define what the agent should do if it runs into this situation or that situation. That’s guardrails. An example is a human-in-the-loop mechanism, which we have in our product Stack Internal. So imagine you send your agent out to perform an action, but we don't know if we have all the answers the agent needs to complete it within our knowledge base. We don't want the agent to act upon knowledge that's incomplete or incorrect.

In Stack Internal, what we do is we label the knowledge as incomplete or incorrect. Then we tell the agent, “If you run into things that are incomplete or incorrect, here's what you should do about it. Don't just use it. Don't assume it's correct. Don't make a judgment call if it's correct or not.” The way we do this is with a trust system. It’ll score knowledge as high, medium, or low. If the score is medium or low, we tell the agent to use a tool called a subject matter expert validation flow. The agent will automatically route its user to an expert on that topic who can validate or fill in blanks for them.

With context architecture, what we're doing is really just removing variables. We remove the variable for an agent that makes it confuse bicycle tires with car tires, and give it boundaries for only sports car tires. We remove the variable of high quality vs. low quality knowledge so it doesn't need to make a judgment call on incomplete or incorrect information. Then we give it a mechanism so if it runs into a wall, it won’t just try to figure it out on its own but instead bring a human-in-the-loop. The whole point is to safely and confidently create tasks for agents with access to knowledge and know that they're going to do the thing you expect them to do. And if they run into a wall, they will also do the thing you expect them to do.

Privacy and security with agent access

Phoebe Sajor: When you give agents this much access to information, how do you keep your data safe? Are there privacy and security concerns with giving agents access to knowledge?

Ash Zade: So, in a lot of cases you'll take a tool like Codex and you’ll connect it to Slack, MS Teams, Google Drive, SharePoint, whatever. And then you'll ask it to do something. The agent is going to pick whichever platform it thinks it should use to find information for your task. But maybe it's going to go into a Slack channel that you don't want it to incorporate data from. It’ll go into an Innovation Slack channel where people are talking about potential things to build and the agent will come back and return facts about something that hasn’t been built yet. So privacy is beyond what the agent should and shouldn’t see. It could simply be that we don't want certain information incorporated into its work at all. That’s why context architecture is so important when giving agents access to data.

When it comes to security, it really becomes about permissions. In Stack Internal, we handle this by giving the agent the same permissions as its source permissions. The user’s access permissions are carried over into Stack Internal. And so, if it's working on my behalf as a product manager, it's going to have the same context as I do and only that context. If I want to restrict it more and say, "Okay, I have access to a lot of things, but I only want you to care about my product area," we have a mechanism called Scopes that allows users to further define what the agent has access to.

So there's two layers here. There are the permissions of the source. If it's a private Slack channel and I'm not a member of it, my agent is not going to be a member either. But even if I am a member of that private Slack channel, I still may not want to incorporate that knowledge into my agent's work. When that happens, I can set a Scope around all the knowledge I have access to and only give the agent a subset of that. So I get to curate the things I have access to, but an agent working on my behalf will never have access beyond what I have access to.

That’s just the retrieval part. But like I said, you want your agents to store memory and create new knowledge context around the things it did. It may be pushing knowledge back into the system. That's where you also want to set some restrictions. You can also do that with Scopes. Okay, so my agent has access to this data, and I want it to capture what it’s doing. But this information is particular to me. I can tell the agent not to put it in the general knowledge pool or share it with my team. Send it to me first. You can control for that, too.

What happens to context when models change

Phoebe Sajor: One of the most interesting things about AI is that everything is constantly changing all the time. It's both exciting and scary. But how does that work with context architecture? What happens to context when the underlying models or systems change?

Doug Whitley: The cool thing about context architecture is it doesn't have to be about how it interacts with a specific AI model. We actually do context architecture in our day-to-day life-you may only keep certain things on your desk because you're trying to keep that context very specific to your workflow. Context architecture is something we naturally engage with day-to-day. We find success when we control the context around us, even physically.

So, the cool thing about context architecture is it’s making things useful regardless of who's on the other end of it. Whether it's an MCP client or server that's doing something for you as a user, whether it's you directly interacting with the context that's being given to you from a search endpoint, or it's just you trying to do some work, keeping a clean context is really valuable. The same is true for AI context architecture, regardless of what AI model you hook up. We're just being explicit about how it might be more useful to codify it and build a tool around it specifically.

How AI agents get fed context through the architecture

Phoebe Sajor: What does the actual process of using context architecture look like in AI systems? How do AI agents get fed context through the architecture?

Doug Whitley: Like anything, it’s going to depend. With Stack Internal, we use a lot of different algorithms to give users the best result. There's indexing, specific tagging that's going to happen, and vector searches you might do that all depend on the question coming in. So part of our process is looking at what's being asked, what's going on in the workflow, and figuring out the best way to search the data.

There’s context mapping, which you can imagine as a crazy board with pins and strings connecting things together. There’s the classical algorithm concept of a table where you have stuff on a table and you can easily grab from it or leave it out. And you can think of Scopes like a stack of sticky notes you’ve written about specific contexts. We also use the trust scores that Ash has talked about previously to figure out what information your agent can safely act on. Stack Internal also re-ranks the information and figures out what context is most important for you and your agent.

Good AI context architecture takes multiple steps. I like to think of it as casting a really wide net, pulling it in, and saying, "Well, we're looking for fish, not oysters. We're going to throw the oysters out. And actually, we only want this kind of fish, so we’ll throw back the other kinds. And we also only want fish this size, so we’ll get rid of ones that are too small or too big.” We’re taking multiple steps to filter out information you don’t need.