Building Your First AI Agent from Scratch (No Framework)

How to build a production-shaped agent in ~150 lines of Python - no LangChain, no CrewAI, nothing you cannot read line by line. A startup founder asked me a question I hear constantly: "Why are we paying for an agent framework when our feature is basically 'call the model, call an API, retry'?" He was not being cheap. He had watched a demo where a framework "did everything," then spent two weeks fighting its abstractions when his workflow did not fit the framework's opinion of how agents should work. The framework was not wrong. It was just a guess about his problem, and his problem was more specific than the guess. So I did what I always do when a framework is in the way: I built the agent by hand. Around 150 lines of Python, no dependencies beyond an OpenAI-compatible HTTP client. When I opened the file for him, every token was traceable - he could see exactly what went into the context, what the model returned, and when the loop decided to stop. He shipped it to production the following week, and it is still running. This article is that build, step by step. You will end with an agent that takes a goal, uses tools, has working memory, respects budgets, and escalates when it is out of its depth - and you will understand every line of it. Once you have built one of these by hand, every framework stops being magic and becomes a set of opinions you can evaluate. Why Build by Hand (When Frameworks Exist) Let me be honest about the trade-off, because there is one. Frameworks like LangChain and CrewAI compress months of patterns into configuration, and for a standard workflow - chat with retrieval, a few tools, an orchestrator - they can genuinely save you a week. The compressed version is also the version you cannot read: when a tool call misbehaves, the stack trace points into the framework's internals, and the framework's memory strategy is a design decision you inherited, not one you made. Building by hand buys you three things you cannot get from configuration: - Readable context. You see every token that enters the model. When the agent behaves oddly, you can reproduce it, because you control the assembly. - Honest budgets. Step limits, cost limits, and escalation are your code, not a flag somewhere in a framework's docs that you may never find. - Debuggable failures. The run log is yours. You know exactly what the agent tried, why it tried it, and where it gave up. The cost is that you write the loop yourself - which is about fifty lines. The rest of this build is tools, memory, and guardrails, which you would write inside a framework anyway. The Architecture We Are Building Before code, the shape of the thing. Our agent runs a loop with four components: ┌─────────────────────────────────────────────┐ │ AGENT LOOP │ │ │ User goal ──▶ assemble context ──▶ model decides ──┐ │ │ │ │ │ │ answer? ──▶ return │ │ │ tool call ─▶ execute ────┼─┘ │ │ │ └─────────────────────────────┼─────────────┘ ▼ budget guards & escalation - Tools are declared functions the model can invoke, executed by our runtime. - Memory is retrieved context injected before the model decides. - Guardrails decide when the loop may continue and when it must stop. - Escalation is a defined hand-off with a readable summary. Step 1: The Tool Layer The first thing to build is the mechanism that turns a model's structured request into a real function call. I keep it boring: a registry of tools, each with a name, a description, a JSON schema, and a Python callable. import json from typing import Callable, Any class Tool: def init(self, name: str, description: str, schema: dict, fn: Callable): self.name = name self.description = description self.schema = schema self.fn = fn def to_openai_spec(self) -> dict: return { "type": "function", "function": { "name": self.name, "description": self.description, "parameters": self.schema, }, } def call(self, arguments: str) -> str: try: args = json.loads(arguments) result = self.fn(**args) return json.dumps(result) except (json.JSONDecodeError, TypeError, KeyError) as exc: return json.dumps({"error": f"invalid tool call: {exc}"}) def lookup_order(order_id: str) -> dict: # Production: query your orders DB here, with authz and caching. return {"order_id": order_id, "status": "paid", "amount": 14900} TOOLS = [ Tool( name="lookup_order", description=( "Look up an order by its ID. Returns status, amount in paise, " "and delivery status. Raises an error if the order does not exist." ), schema={ "type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"], }, fn=lookup_order, ), ] Two details matter. The description is a contract for the model, not a comment for humans - the model reads it to decide when to call this tool, so it must state preconditions. And the call() method never lets an exception crash the loop; a bad call becomes a readable tool result that the model can recover from. Step 2: Memory as Retrieved Context Next, memory. I am going to keep this minimal but production-shaped: an in-memory store of past facts, retrieved by the model's own sense of relevance via a vector index. For a first agent, you do not need a database server; you need the layer, and you need it to be the thing you swap later. class SimpleMemory: def init(self, embed: Callable[[str], list[float]]): self.embed = embed self.items: list[tuple[str, list[float]]] = [] def remember(self, text: str) -> None: self.items.append((text, self.embed(text))) def recall(self, query: str, top_k: int = 3) -> str: if not self.items: return "(no memory yet)" q = self.embed(query) scored = sorted( self.items, key=lambda item: _cosine(item[1], q), reverse=True, ) return "\n---\n".join(text for text, _ in scored[:top_k]) For the embedding function, use any OpenAI-compatible endpoint with text-embedding-3-small or a local model. In production this becomes pgvector or Qdrant; the interface - remember() and recall() - is what survives the swap. That is the real value of building the layer yourself: you own the seam. Step 3: The Loop with Guardrails Now the heart. The loop assembles context, calls the model, and interprets the response. The guardrails are not an afterthought here; they are written into the loop itself so they cannot be skipped. from openai import OpenAI class Agent: def init(self, model: str, tools: list[Tool], memory: SimpleMemory, client: OpenAI): self.model = model self.tools = {t.name: t for t in tools} self.memory = memory self.client = client def run(self, goal: str, max_steps: int = 6, max_cost_cents: float = 10.0) -> dict: system = ( "You are an assistant that completes a goal using available " "tools. Rules: only call a tool when you need data; never invent " "tool results; if you cannot finish, escalate with a summary of " "what you tried and what is missing." ) history = self.memory.recall(goal) messages = [ {"role": "system", "content": system}, {"role": "user", "content": f"RELEVANT PAST CONTEXT:\n{history}\n\nGOAL: {goal}"}, ] steps = 0 cost = 0.0 for steps in range(1, max_steps + 1): response = self.client.chat.completions.create( model=self.model, messages=messages, tools=[t.to_openai_spec() for t in self.tools.values()], ) cost += _estimate_cost(response) # tokens * rate if cost > max_cost_cents: return {"outcome": "escalated", "summary": "cost budget exceeded", "steps": steps, "cost_cents": cost} msg = response.choices[0].message if not msg.tool_calls: self.memory.remember(f"goal: {goal} -> answer: {msg.content}") return {"outcome": "done", "answer": msg.content, "steps": steps, "cost_cents": cost} messages.append(msg) for call in msg.tool_calls: tool = self.tools.get(call.function.name) result = ( tool.call(call.function.arguments) if tool else json.dumps({"error": "unknown tool"}) ) messages.append({ "role": "tool", "tool_call_id": call.id, "content": result, }) return {"outcome": "escalated", "summary": "step budget exhausted, tried: " + repr([m.get("content", "")[:80] for m in messages[-4:]]), "steps": steps, "cost_cents": cost} Read the guardrails, because they are the product: a step budget so the loop cannot run forever, a cost budget that fails soft, unknown-tool handling so a hallucinated tool name cannot crash the run, and a memory write-back so the next goal starts from what this run learned. Escalation returns a readable summary - that is what the human receives, not an error trace. Step 4: Wiring It Together def embed(text: str) -> list[float]: r = client.embeddings.create(model="text-embedding-3-small", input=text) return r.data[0].embedding client = OpenAI() # any OpenAI-compatible endpoint memory = SimpleMemory(embed=embed) agent = Agent(model="your-model", tools=TOOLS, memory=memory, client=client) result = agent.run("Where is order ORD-9911 and has it been paid?") print(result) Run that and the agent will call lookup_order , get the status, and answer - or escalate if it cannot. That is the entire skeleton. It is small because the loop is small. Everything you will ever add - a vector database, retries, multiple agents, a permission layer - attaches to one of the seams you just built. Production Reality: What I Broke on My First Build I have built enough of these to know the exact ways the naive version fails, and you will hit them too. Here is the list in the order they will find you: 1. The tool description was too lazy. My first lookup_order said "gets order info," and the model called it for users' own made-up order IDs. Rewriting the description to state preconditions and error behavior fixed most misuse without any code change. Treat descriptions as documentation, because that is what the model reads. 2. Malformed JSON killed the run. The model occasionally emitted truncated JSON arguments, and my first version threw, killing the loop. The Tool.call() error path - returning a readable error instead of raising - is what saved it. A model that gets a tool error can recover; a loop that crashes cannot. 3. The cost guard was the first thing I removed, and I regrette