🧠 Tokenization in Modern LLMs: The Hidden Mechanics Behind AI Costs and Reasoning

Tokens: The Fundamental Unit of Language Models

Large language models do not process text as humans do. They do not see full words or letters in isolation. Instead, they operate on tokens, which are compressed chunks of text. A token can be:

• A full word (“error”)
• A word fragment (“stra”, “wberry”)
• A punctuation mark (“.”)
• Or even whitespace patterns

Think of text as a construction kit where tokens are modular components. Instead of building language from raw characters or entire words, models assemble meaning from pre-shaped linguistic “blocks”.

Byte Pair Encoding (BPE): How Tokens Are Created

At the heart of modern tokenization sits Byte Pair Encoding, a statistical compression method adapted for language. Here is the intuition:

• Start with every character as a separate unit
• Scan massive text datasets
• Identify the most frequent adjacent character pairs
• Merge them into new tokens
• Repeat millions of times

Over time:

• Common patterns become single tokens (“the”, “ing”, “tion”)
• Rare words remain split into multiple pieces
• Code, typos, and unusual strings get fragmented further

This is why language models are efficient: they learn a reusable vocabulary of patterns instead of memorizing every possible word.

The Cost Reality: Tokens Are Money

One of the most important engineering truths: You are not billed per word. You are billed per token.

A rough heuristic:

• 1 token ≈ 0.75 words (English average)

But inefficiencies in formatting can inflate token usage quickly. Even small differences matter:

• “Fix the error.”
• “Fix the error .”

That extra space and punctuation irregularity can cause additional token splits, increasing cost at scale.

Token Economics: Input vs Output

In production systems, token usage splits into two categories:

• Input tokens: what you send to the model
• Output tokens: what the model generates

Output tokens are usually more expensive because they require sequential generation and higher compute workload.

Example pricing structure (illustrative):

This is why architectural decisions in prompts and payload formatting directly translate into infrastructure spend.

Formatting Matters: JSON Isn’t Free

Even formatting becomes a cost variable.

Pretty JSON:

{
  "status": "unhealthy",
  "replica_count": 0
}

Minified JSON:

{"status":"unhealthy","replica_count":0}

At scale, the difference is significant:

• Pretty format introduces whitespace tokens and line breaks
• Minified format reduces token count dramatically

In real systems, minification can reduce token usage by 30–40% for structured payloads.

The “Strawberry Problem”: Why Models Miscount Letters

A famous demonstration shows models failing to count letters in “strawberry”. This is not a math failure. It is a visibility limitation.

When a human sees:

strawberry → s t r a w b e r r y

A model sees something like:

["straw", "berry"]

Inside the tokenizer:

• The word is split into semantic chunks
• Individual letters are no longer directly accessible

So when asked: “How many R’s are in strawberry?” The model effectively cannot inspect the raw spelling. It only reasons over token fragments:

• “straw” → no visible letter-level structure
• “berry” → statistical association of letters, not explicit decomposition

Result: the reasoning becomes approximate rather than literal. This phenomenon is often called token blindness.

Vocabulary Size: Why GPT-4o Became More Efficient

Earlier tokenizers (like cl100k_base) used ~100K token vocabularies. With newer systems such as GPT-4o, vocabulary size expanded to ~200K tokens.

Impact:

• More compact encoding of multilingual text
• Better handling of non-English languages
• Reduced token fragmentation in code and structured data

For developers, this means:

• Lower token consumption per request
• Faster inference in many cases
• Reduced cost in global applications

Special Tokens: Invisible Control Signals

Not all tokens represent human-readable text. Some act as structural commands inside the model pipeline:

• <|endoftext|> → marks end of input stream
• <|im_start|> / <|im_end|> → separates system, user, and assistant roles

These are critical for:

• Conversation structure
• Instruction hierarchy
• Safety boundaries

However, if manipulated incorrectly, they can contribute to prompt injection-style vulnerabilities, where system boundaries are confused or bypassed.

Beyond Tokens: What Comes Next

Understanding tokens unlocks deeper concepts in LLM engineering.

Token IDs

Tokens are converted into integers before entering the model:

• “the” → 464 (example ID)
• “error” → 3921 (example ID)

The model only ever sees numbers.

Context Window Limits

Every model has a fixed memory window:

• Input tokens + output tokens must fit inside it
• Exceeding it causes truncation or loss of earlier context

This is why long conversations “forget” earlier details.

Rare or “Glitch” Tokens

Because tokenizers are trained on massive web corpora, they sometimes include:

• obscure strings
• leaked identifiers
• unusual code fragments

These can behave unpredictably in edge cases, especially in adversarial inputs.

Final Takeaway

Tokenization is not just a preprocessing step. It is the economic, structural, and cognitive backbone of modern language models. Once you understand tokens:

• Prompt engineering becomes predictable
• Cost modeling becomes transparent
• Model limitations become explainable rather than mysterious

Credit

This rewritten technical synthesis is based on the original educational work and concepts by Naveen Kumar VR, from his MLOps and AI series on language model fundamentals.