Phase 7a - Getting Opinionated: Rules-Based Auto-Categorization (and a Seam for the AI Later)

My expense app finally has a point of view on what I'm spending money on. No AI yet - just honest keyword rules, a nullable column, and one interface that means I can bolt an LLM on later without ripping anything out. Here's the build, three "empty value" bugs that bit me, and the habits that kept it clean. Index - Where we left off - The plan: rules first, AI behind the same door - Step 1 - A nullable column (and why nullable matters) - Step 2 - The migration: generate → review → apply - Step 3 - A dumb-but-working categorize() - Step 4 - Wiring it into create (with override precedence) - Step 5 - The seam: extracting behind a Categorizer interface - Step 6 - The UI loop: show, add, edit - 🐛 The war story: three ways "empty" lied to me - Thinking like an attacker - Learning shortcut vs. production - Key habits to keep - Next up: Phase 7b Where we left off Phase 6 gave me the receipts - date-range reports and CSV export. I ended that post with a promise: Next up: Phase 7, where categories finally enter the schema and the app starts to get opinionated about what I'm spending on. This is that. But it turned into a bigger beast than one post, so I'm splitting it: - Phase 7a (this post): the schema, a rules-based categorizer, the interface seam, and the full UI loop. - Phase 7b (next): the actual LLM - an LLMCategorizer that slots in behind the same interface, with caching and a rules fallback. Doing rules first isn't a cop-out. It's the whole strategy. The plan: rules first, AI behind the same door The temptation with "AI categorization" is to reach straight for the API key. I didn't. Here's the order I actually built in, and why: | Step | What | Why this order | |---|---|---| | 1 | Nullable category column | The app needs somewhere to store a category before it can fill one | | 2 | Rules categorize() | A working, free, offline fallback - and a baseline to test against | | 3 | Extract behind an interface | So the LLM can slot in later without touching call sites | | 4 | UI loop (show / add / edit) | Give the human final say, no matter how smart the auto-fill gets | | 5 | (Phase 7b) LLM implementation | The risky external dependency goes in last, behind the seam | The principle: build the boundary before you plug in the flaky thing. Rules are boring and reliable. The LLM will be clever and occasionally down, slow, or wrong. If both live behind the same contract, swapping between them - or falling back - is a one-line change. Step 1 - A nullable column (and why nullable matters) One new line on the Expense model: category: Mapped[str | None] = mapped_column(String(50), nullable=True) The interesting bit is the | None / nullable=True . Every expense already in my database was created before this column existed. If I made it nullable=False , the migration would try to force a value into all those existing rows and fail - or demand a default I don't actually want. Nullable is honest: old rows are genuinely uncategorized. "No category" is a real state, not an error. And it maps perfectly onto the rules engine returning "I don't know" - more on that below. New syntax I picked up here: | Syntax | Meaning | |---|---| Mapped[str] | A required string | | Mapped[str \ | None] | String(50) | Length-bounded, like my description at 255 | Step 2 - The migration: generate → review → apply Three deliberate moves, not one: # 1. Generate (does NOT touch the DB - just writes a script) alembic revision --autogenerate -m "add category column to expenses" # 2. Review - open the file, confirm it does EXACTLY one thing # upgrade() -> add_column('expenses', 'category', String(50), nullable=True) # downgrade() -> drop_column('expenses', 'category') # reversible! # 3. Apply (this one DOES modify the database) alembic upgrade head Autogenerate is good, not infallible. It occasionally invents spurious type tweaks, and SQLite has its quirks with certain operations. So I read the file every single time before applying - confirming both that upgrade() does only what I asked and that downgrade() cleanly reverses it. Backup habit: SQLite is a single file, so a backup is a cp : cp expenses.db expenses.backup.db alembic upgrade head Verified the column actually landed: python -c "import sqlite3; print(sqlite3.connect('expenses.db').execute('PRAGMA table_info(expenses)').fetchall())" # ...a 'category' row, VARCHAR(50), nullable. Existing rows: None. (Aside: my .gitignore had *.db but not *.bak . Named the backup expenses.backup.db so the existing rule caught it, then added *.bak anyway as housekeeping.) Step 3 - A dumb-but-working categorize() No AI. Just keywords: CATEGORY_RULES = { "Food": ["swiggy", "zomato", "restaurant", "cafe", "coffee", "pizza", "lunch", "dinner", "breakfast", "meal", "food", "grocery", "canteen"], "Transport": ["uber", "ola", "cab", "fuel", "petrol", "metro"], "Shopping": ["amazon", "flipkart", "myntra", "mall"], "Utilities": ["electricity", "water", "gas", "internet", "wifi", "recharge"], "Entertainment": ["netflix", "spotify", "movie", "bookmyshow"], } def categorize(description: str) -> str | None: text = description.lower() for category, keywords in CATEGORY_RULES.items(): if any(keyword in text for keyword in keywords): return category return None The concepts I actually learned writing this: | Concept | What it does | |---|---| .lower() | Case-insensitive matching - "UBER" , "Uber" , "uber" all hit | keyword in text | Substring test - "uber" in "uber to airport" is True | any(... for ...) | Returns True on the first match, then stops | return None | No match = no guess - maps onto the nullable column | That last line is the design decision I'm proudest of. When the rules don't recognize something, they don't guess wildly - they return None , and the expense stays honestly uncategorized. Tested in isolation before wiring anything: python -c "from main import categorize; print(categorize('Uber to airport'), '|', categorize('Swiggy dinner'), '|', categorize('Random mystery charge'))" # Transport | Food | None Step 4 - Wiring it into create (with override precedence) The rule: if the user explicitly sends a category, respect it. Only auto-fill when they don't. data = payload.model_dump() if data.get("category") is None: data["category"] = categorize(data["description"]) expense = Expense(**data, user_id=current_user.id) data.get("category") returns None if the key is missing or sent as null - my "did the user leave it blank?" test. Blank → auto-categorize. Provided → keep theirs. This is also where the first landmine was waiting (see the war story). But conceptually: the client now has three honest options - omit it (auto), send a value (override), or send null (explicitly uncategorized). Step 5 - The seam: extracting behind a Categorizer interface This is the step that makes Phase 7b painless. Right now categorize() lives inside a web route - core domain logic welded to the HTTP layer. That's a smell, and it blocks the LLM work. So I pulled it into its own module behind a contract: # backend/categorization.py from typing import Protocol class Categorizer(Protocol): """The contract every categorizer must satisfy.""" def categorize(self, description: str) -> str | None: ... class RulesCategorizer: """Keyword-rules implementation of the Categorizer contract.""" def init(self, rules: dict[str, list[str]] = CATEGORY_RULES): self._rules = rules def categorize(self, description: str) -> str | None: text = description.lower() for category, keywords in self._rules.items(): if any(keyword in text for keyword in keywords): return category return None # The single instance the rest of the app imports. default_categorizer: Categorizer = RulesCategorizer() Three ideas clicked here: - A Protocol is a contract. "Anything called aCategorizer must havecategorize(description) -> str | None ." A class satisfies it just by having that method - no inheritance (structural typing). My futureLLMCategorizer will satisfy the same contract for free. - A class carries state. A bare function can't remember things. The LLM version will need to - an API client, and eventually a per-merchant cache. So a class, with self._rules on the instance. - One entry point. Everything imports default_categorizer and calls.categorize(...) , blissfully ignorant of how it works. Swapping rules → LLM becomes a one-line change in one place. Then main.py just does: from categorization import default_categorizer # ... data["category"] = default_categorizer.categorize(data["description"]) Refactor discipline: I shipped the extraction as a pure, behavior-neutral commit - same inputs, same outputs - and only then, in a separate commit, widened the Food keywords. Never mix a refactor with a behavior change in the same commit. When something breaks later, you want git bisect to land on one or the other, not a tangle of both. Step 6 - The UI loop: show, add, edit Auto-categorization is useless if the human can't see or override it. Three small React slices: Show - a category pill per row, with a loud fallback for the ones the rules missed: {expense.category ? ( {expense.category} ) : ( Uncategorized )} I made the "Uncategorized" pill red on purpose - it's a gentle nudge that says "this one still needs you." Add - an optional category input on the create form, plus defaulting the date to today. Edit - the same override on the inline edit form, so those red flags are actionable: click, type a category (or clear it), save. Both forms share one trick that turned out to be load-bearing - which brings me to the bugs. 🐛 The war story: three ways "empty" lied to me Phase 6 had one signature bug (a CSV that only wrote its last row, traced to an indentation slip). Phase 7a had a theme: every single bug this session was about the difference between null, empty string, omitted, and the wrong day. "Nothing" is not one thing. It's at least four, and they don't behave the same. Gotcha 1 - Nullable ≠ optional (the silent 422) I declared the create field like this and thought I was done: #