How to get more from your chatbot for less [P]

Stop Tokenmaxxing. Start Tokenminning.

This article contains real-world patterns you can use to minimize your API/Agent costs (without major refactoring).

Instead of just glazing over routing, it provides a real-world example of routing with a pretrained classifier and an actual routing table which works. It also provides a recipe for training your own prompt classification model.

Use this for cost reductions of up to 60%.

The Core Strategy: Intelligent Routing

The key insight is that not every user request needs to hit your most expensive, most capable model. By classifying incoming prompts and routing them to the cheapest adequate model, you can dramatically cut costs.

How Routing Works

1. A pretrained classifier analyzes each incoming prompt.
2. The classifier assigns the prompt to a category (e.g., "simple Q&A," "code generation," "creative writing").
3. A routing table maps each category to the cheapest model that can handle it adequately.
4. The prompt is sent only to the assigned model.

Example Routing Table

Training Your Own Prompt Classification Model

You can train a lightweight classifier specifically for your use case. Here's a recipe:

Step 1: Collect Training Data

Gather examples of prompts your users send, labeled with the correct routing category. Aim for at least 500 examples per category.

Step 2: Choose a Base Model

Use a small, efficient model like distilbert-base-uncased or microsoft/deberta-v3-small.

Step 3: Fine-Tune the Classifier

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)

# Your training data here
train_encodings = tokenizer(train_texts, truncation=True, padding=True)
train_labels = train_labels

training_args = TrainingArguments(
    output_dir="./classifier",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    save_steps=500,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

Step 4: Deploy the Classifier

Serve the trained model as a lightweight API endpoint or embed it directly in your application.

Real-World Results

Teams implementing this pattern report:

• 40-60% reduction in API costs
• No degradation in response quality for 90%+ of queries
• Faster response times for simple queries routed to smaller models

The classifier itself costs less than $0.001 per classification, making the savings substantial at scale.