How we optimized our LLM pipeline to cut token usage by 70%

Most teams assume the fastest way to reduce AI costs is to switch to a smaller model. In reality, that's often the last thing you should do.

Within a few weeks we noticed three problems:

• API costs were increasing every day.
• Response latency became inconsistent.
• Most requests didn't actually need the most capable (and most expensive) model.

Downgrading to a cheaper model reduced costs, but it also reduced answer quality. Instead, we optimized the entire pipeline. The result was a significant reduction in token usage and API spend while keeping response quality almost unchanged. Here's what made the biggest difference.

Stop sending everything to the biggest model

One of the biggest cost drivers wasn't the model itself - it was how we used it. Tasks like summarizing text, extracting keywords, formatting JSON, or rewriting an email don't require advanced reasoning. Running these requests on your most expensive model is like using a supercomputer to open a spreadsheet.

Instead, introduce a lightweight routing layer before calling the LLM.

User Request
    │
    ▼
┌────────────────────┐
│ Complexity Router  │
└────────────────────┘
    │               │
Simple Task    Complex Task
    │               │
    ▼               ▼
GPT-4.1 Mini    GPT-5

A simple classifier-or even a few rule-based checks-can decide which model is appropriate. This optimization alone can dramatically reduce API costs without affecting the user experience.

Shorter prompts are usually better prompts

Long prompts are expensive. Unfortunately, prompts tend to grow over time. Every new feature adds another instruction, another example, or another formatting rule. Eventually, the prompt becomes larger than the user's actual question.

The shorter version is easier to maintain, consumes fewer tokens, and often produces the same quality of responses. Instead of continuously adding instructions, periodically review your prompts and remove anything that no longer provides measurable value.

Rule of thumb: Every token in your prompt is a token you pay for.

Add Semantic Caching

Traditional caching only works when two requests are identical. LLM applications rarely receive identical requests. For example: "How do I request my password?" vs. "I forgot my password". The wording is different, but the intent is the same. This is where semantic caching becomes useful.

User Question
    │
    ▼
Generate Embedding
    │
    ▼
Vector Similarity
    │
┌────────┴─────────┐
│                  │
Cache Hit      Cache Miss
│                  │
▼                  ▼
Cached Response  Call LLM

Instead of comparing text character by character, compare the meaning of each request. If a sufficiently similar response already exists, return it immediately instead of calling the LLM again. For customer support, internal knowledge bases, or FAQ assistants, semantic caching can eliminate a large percentage of duplicate requests.

Send less context to the model

A common misconception is that more context always leads to better answers. In reality, too much context often has the opposite effect. Many RAG pipelines simply retrieve the top 10 or 20 document chunks and send everything to the model.

Knowledge Base
    │
Retrieve Top 10 Chunks
    │
    ▼
    LLM

This increases token usage, latency, and cost. Instead, retrieve broadly, then filter aggressively.

Knowledge Base
    │
Hybrid Search
    │
Reranker
    │
Top 3 Chunks
    │
    ▼
    LLM

The model receives fewer tokens, but those tokens are significantly more relevant. In many cases, this not only reduces costs but also improves answer quality by removing distracting information.

Ask for structured output

Structured outputs offer several advantages:

• fewer output tokens
• faster responses
• easier parsing
• more reliable downstream automation

Sometimes we accidentally ask the model to generate much more text than we actually need. Imagine building a travel booking platform. If your application only needs destination, travel dates, and budget, asking the model to "explain its reasoning" wastes both time and tokens.

Instead of this:

Explain your reasoning before generating the final answer.

Ask for structured data directly.

{
  "destination": "Tokyo",
  "duration": "5 days",
  "budget": "$1500"
}

Final thoughts

Optimizing an LLM application isn't just about choosing the right model-it's about designing a smarter pipeline around it. In our experience, the biggest cost savings didn't come from switching to a smaller model. They came from reducing unnecessary tokens, routing requests more intelligently, caching repeated queries, and providing only the context the model actually needed.

As LLM applications move from prototypes to production, these optimizations become increasingly important. A few architectural improvements can significantly reduce costs, improve latency, and create a better user experience-all without compromising response quality.

Start by measuring where your tokens are going. You'll likely find that the biggest opportunities aren't inside the model, but in everything around it.