Building a Ride-Share Zone-Balancing Agent with LangGraph - Part 2: Teaching the Agent to Read Ops Notes

This is Part 2 of a 5-part series. Part 1 built a rule-based agent for a single ride-share zone - no LLM, just structured numbers and deterministic rules. It worked, but with one obvious hole. An ops person doesn't type structured fields. They write something like "the event got cancelled, and there's an accident backing up traffic near the venue." Stage 1's agent only understands rain_flag /event_flag /traffic as raw values. It has no way to read that sentence at all. This is where an LLM earns its place - for exactly two jobs a lookup table genuinely can't do, and nothing more: - Reconcile an ops note. Read the free text and correct rain_flag /event_flag /traffic before the decision runs. - Explain the decision. Write a plain-English justification for a human reviewer, after the policy is already chosen. Policy selection itself doesn't change at all. It's the exact same deterministic logic from Part 1, reused directly - never redone, never second-guessed by the LLM. A small local model isn't automatically trustworthy, even at a narrow job like this. So I also compare five of them side by side, to see which one actually holds up. Why Not Just Hand the LLM Everything? An earlier version of this project tried that. It used a ReAct loop, where the LLM called several tools and picked the final policy itself, via free-text output. It didn't hold up. Even qwen2.5:14b sometimes compared the wrong pair of candidates, or dropped the best option from its own answer. That wasn't a prompt-tuning problem - it was the wrong tool for the task. An LLM doesn't compute an argmax. It generates plausible-sounding text, with no guarantee every candidate actually got compared. max() never makes that mistake. Here's the rule I use throughout this project: if a task can be solved with plain numbers and a fixed answer, write it in code. Code gets it right every time. An LLM only earns its place when the input is a sentence a lookup table can't parse, or the output has to be a sentence a lookup table can't write. Handing a plain numeric decision to an LLM doesn't add capability - it just adds a new way to be wrong. reconcile_inputs reads a real sentence, so that's free text in. generate_explanation writes one, so that's free text out. Picking the most profitable policy out of four numbers is neither, so it stays in plain Python: # Scenario: 4 drivers, 14 rider requests, event nearby for policy in ["surge_pricing", "driver_bonus", "demand_redirect", "do_nothing"]: r = evaluate_policy({"driver_count": 4, "rider_request_count": 14, "rain_flag": False, "event_flag": True}, policy, noise=False) print(f"{policy: 8.2f} resolved={r['resolved']}") surge_pricing profit=$ 75.61 resolved=False driver_bonus profit=$ 56.66 resolved=False demand_redirect profit=$ 62.96 resolved=False do_nothing profit=$ 51.60 resolved=False Same evaluate_policy function as Part 1, called directly. No LLM anywhere near it. The Graph Two single-shot LLM calls get added, and nothing else changes: detect_imbalance ──▶ classify_severity ──▶ set_candidates │ ┌────────────┴────────────┐ ▼ (balanced) ▼ (deficit/surplus) trivial_do_nothing reconcile_inputs ← LLM #1 │ ▼ │ resolved_imbalance │ ▼ │ choose_best_policy │ ▼ │ generate_explanation ← LLM #2 │ │ └──────────────┬───────────┘ ▼ simulate_and_report detect_imbalance , classify_severity , resolved_imbalance , and choose_best_policy are imported directly from Part 1's module. Same deterministic logic - not redone here. The two new nodes are single-shot, not a loop: - reconcile_inputs only runs if there's an ops note. It calls a tool, but only to get structured data back. The tool's code never actually runs - calling it is just how the model reports its answer, instead of writing a paragraph. - generate_explanation runs afterchoose_best_policy has already decided. It's plain text, no tools. It can't get the decision wrong, because it never touches it. Balanced zones skip both entirely. One candidate, nothing to reconcile or explain - zero model calls. The State Same idea as Part 1's state. Three fields get added, for what an LLM now contributes: - ops_note - the input - explanation - an output - messages - the raw LLM conversation, kept around for inspection class AgentState(TypedDict): zone: dict ops_note: str imbalance_ratio: float imbalance_type: str severity: str candidate_policies: list policy_evaluations: dict policy_resolutions: dict recommended_policy: str explanation: str messages: Annotated[list[BaseMessage], operator.add] outcome: dict outcome_delta: dict report: str Structured Output via bind_tools reconcile_inputs doesn't let the model write back a free-text paragraph. Instead, it gives the LLM exactly one tool, and asks it to call that tool once - with the three fields it needs to report: @tool def report_context(rain_flag: bool, event_flag: bool, traffic_level: Literal["none", "light", "moderate", "heavy"]) -> str: """ Call this exactly once with your interpretation of rain_flag, event_flag, and traffic_level for this decision, after considering the ops note. If nothing in the note changes a value, report the zone's original value unchanged. traffic_level anchors: "none" = no congestion mentioned/normal; "light" = some congestion, minor delays; "moderate" = noticeable backup, routes slower than usual; "heavy" = gridlock, accident, or road closure - significant delays. Only road/vehicle traffic counts here - foot traffic (pedestrian crowds) is a separate concept and should NOT affect this value. """ return f"rain_flag={rain_flag}, event_flag={event_flag}, traffic_level={traffic_level}" The tool's body never actually runs. Calling it is just a clean way for the model to hand back exactly three values: two booleans and one fixed label. That beats a paragraph I'd have to parse by hand. Wiring It Up Every node here is a plain function, same as Part 1. reconcile_inputs and generate_explanation just take an extra llm argument. So building the graph means binding a specific model first. I wrap this in a build_agent(llm) function, instead of building it inline once. Here's why: the model comparison below needs the exact same graph, rebuilt five separate times, once per LLM. The structure never changes. Only the model bound to it does. def build_agent(llm): llm_with_reconcile_tool = llm.bind_tools([report_context]) def _reconcile(state): return reconcile_inputs(state, llm_with_reconcile_tool) def _explain(state): return generate_explanation(state, llm) g = StateGraph(AgentState) g.add_node("detect_imbalance", detect_imbalance) g.add_node("classify_severity", classify_severity) g.add_node("set_candidates", set_candidates) g.add_node("trivial_do_nothing", trivial_do_nothing) g.add_node("reconcile_inputs", _reconcile) g.add_node("resolved_imbalance", resolved_imbalance) g.add_node("choose_best_policy", choose_best_policy) g.add_node("generate_explanation", _explain) g.add_node("simulate_and_report", simulate_and_report) g.add_edge(START, "detect_imbalance") g.add_edge("detect_imbalance", "classify_severity") g.add_edge("classify_severity", "set_candidates") g.add_conditional_edges("set_candidates", route_llm_or_skip, { "trivial_do_nothing": "trivial_do_nothing", "reconcile_inputs": "reconcile_inputs", }) g.add_edge("reconcile_inputs", "resolved_imbalance") g.add_edge("resolved_imbalance", "choose_best_policy") g.add_edge("choose_best_policy", "generate_explanation") g.add_edge("generate_explanation", "simulate_and_report") g.add_edge("trivial_do_nothing", "simulate_and_report") g.add_edge("simulate_and_report", END) return g.compile() One Ops Note, Five Models Policy selection can't go wrong here. It's still deterministic. What actually varies from model to model is narrower: whether it correctly reads the ops note and reports the corrected event_flag and traffic_level . The scenario: Stadium Area's raw snapshot has event_flag=True and no traffic. The ops note says the event was cancelled, and separately describes an accident backing up the main road. That's two corrections. It also has a trap phrase - "foot traffic" - that shouldn't affect traffic_level at all: OPS_NOTE = ( "The stadium event just got cancelled - foot traffic will be much lower than " "usual tonight. There's also a bad accident on the main road backing up traffic " "near the venue." ) A model reading this correctly reports event_flag=False and traffic_level="heavy" . And it doesn't let "foot traffic" leak into that second value. I ran four 7-8B models first - the range most people can comfortably run on a laptop. Then I moved up to qwen2.5:14b , to see what the extra parameters buy: | Model | Reconciled correctly? | Notes | |---|---|---| deepseek-r1:8b | No | Never called the tool at all - not this run, not any run so far | llama3.1:8b | Yes (this run) | In an earlier run, returned 'False' as a string, not a boolean - bool("False") is True in Python | qwen2.5:7b | Yes (this run) | Documented as inconsistent across repeated testing - sometimes right, sometimes wrong, occasionally empty | mistral:7b | No (this run) | Got it right in an earlier run; didn't call the tool at all this time | qwen2.5:14b | Yes | Correct on every run so far | The pattern that matters isn't any single row. It's that llama3.1:8b and mistral:7b swapped which one failed, between two runs of the identical notebook - same prompt, same temperature=0 . That's concrete proof that one clean run isn't evidence of reliability. deepseek-r1:8b is the one exception that looks systematic, not random. It never calls the tool at all, on any run. Its reasoning step doesn't reliably turn into structured tool output. qwen2.5:14b (~9GB, still comfortable on a 16GB M2) is the only model that's held up correctly on every run of this notebook. Model choice mattered more than prompt tuning here - even though the task itself is scoped tight: read two or three sentences, report two booleans and one bounded label. Final model for this project: qwen2.5:14b . Every later part defaults to it. Not because it's