Why message 50 of your chatbot costs 19x more than message 1
You build a chatbot and quote the cost of one short message. The real bill arrives, and it is far higher. The reason is two ideas: tokens, and the context window. Here is the arithmetic. All numbers below are made up for illustration.
What is a token?
A model does not read whole words. It reads tokens: a short word, part of a long word, a piece of a number, or a punctuation mark. A tokenizer cuts your text into these pieces before the model sees it.
- Rule of thumb for English: one token is about four characters, so a 500-word page is roughly 700 tokens.
- Code, numbers and non-English text need more tokens for the same meaning, so they cost more.
- Repeated words are counted every time. You pay for both copies.
- Each model has its own tokenizer, so the same text has different counts in different models. Use the AI company's own counter when the count matters.
What is a context window?
Think of it as the model's desk, counted in tokens. Everything for one request must fit on it at once:
- The system prompt
- The chat so far
- Your documents
- Your question
- Room for the answer
If the total is too big, the request fails with a length error. The model also keeps nothing between requests, so the app fills the desk again every time.
Worked example: message 50 vs message 1
Because the app re-sends the history with each new message, a long chat gets heavier with every turn. Assume:
- The system prompt is 500 tokens.
- Each turn adds 200 tokens: a 40-token message plus a 160-token reply.
Message 1 sends: 500 + 40 = 540 tokens
Before message 50 there are 49 old turns: 49 x 200 = 9,800 tokens
Message 50 sends: 500 + 9,800 + 40 = 10,340 tokens
Compare them: 10,340 / 540 = about 19
The customer types about the same amount each time. The same short question sends about 19 times more input, and input is billed per token on every request.
The same pattern at other points in the chat: 2,340 tokens at message 10, and 5,340 at message 25.
What to do about it
A bigger window only delays the problem, because cost and waiting time grow with the size of what you send. Send less, but better:
- Retrieval: search first, then send only the few best passages instead of everything.
- Trim history: keep recent turns, replace older ones with a short summary, and drop what the next answer does not need.
- Budget each category (system prompt, history, documents, question, answer) separately. The bill shows only a total, and the total does not tell you what to cut.
- Plan for your longest chats, not the average one. History is re-sent, so a few very long chats cost the most.
Caveats
- The numbers here are made up. Your system prompt, turn length and window size will differ.
- Real tokenizers cut text in other places than any simple rule. The rule of thumb is a rough guide.
- Replies often cost more per token than input. Check your provider's price list rather than assuming.
- Reserving room for the answer matters. If input plus the reserved answer is bigger than the window, something must be trimmed.
Go deeper
This article is a short version of Chapter 7 of Deployed Lab. The interactive chapter has a tokenizer you can type into, a slider tool to spend a 32,000-token window across five parts, and a short quiz. It is free with no signup: Tokens and context windows. Deployed Lab is a course on AI engineering and the forward deployed engineer role.
Comments
No comments yet. Start the discussion.