Count tokens before you call the LLM: tiktoken encodings, token budgets and chunking, with real numbers
DEV Community

Count tokens before you call the LLM: tiktoken encodings, token budgets and chunking, with real numbers

Thursday AI Tool/Workflow. This week's tool: tiktoken, OpenAI's open-source BPE tokenizer. It turns text into the same token IDs that OpenAI models see, so you can count prompt tokens before you send the request.

Use it to check that the prompt fits the context window, cut documents by tokens instead of characters, and estimate what a call will cost. Most "context length exceeded" errors and surprise bills come from guessing "about 4 characters per token."

On my samples that guess was right for none of them. English came out at 6.0 characters per token, while a Python snippet, a JSON payload and a Hindi sentence came out at 2.5-2.8.

Here's the workflow, with a script I ran on the box, the real output, and the gotchas I hit.

What it is

tiktoken is a fast byte pair encoding (BPE) tokenizer for OpenAI's models. The core is written in Rust and wrapped in Python. Per the README, BPE is reversible and lossless, works on any text (even text not in the tokenizer's training data), and compresses text, with each token covering "about 4 bytes" on average in practice.

The thing to get right is the encoding. Different models use different vocabularies:

  • o200k_base: the newer one. encoding_for_model() maps gpt-4o, gpt-4o-mini, gpt-4.1, gpt-5 and the o1 / o3 / o4-mini families to it.
  • cl100k_base: gpt-4, gpt-3.5-turbo and the text-embedding-3-* / ada-002 embedding models.
  • p50k_base and r50k_base: older Codex and GPT-3 models.

The OpenAI Cookbook table lists them all.

1. Install

python -m venv venv && source venv/bin/activate
pip install tiktoken

I used tiktoken 0.14.0, the current release on PyPI and GitHub (published August 17, 2026, MIT license, Python 3.9+). The changelog for 0.14.0 says it builds wheels for Python 3.15 and supports looking up more GPT-5 series models.

My box is an 8-vCPU Intel Xeon with no GPU. Tokenizing doesn't need one.

2. The workflow, in code

The script below does seven things:

  1. Picks the encoding with get_encoding() and encoding_for_model().
  2. Counts tokens for four kinds of text under both encodings.
  3. Checks the encode/decode round-trip and looks at a few token pieces.
  4. Shows how special tokens are handled.
  5. Truncates and chunks text by tokens.
  6. Budgets a chat prompt and estimates its cost.
  7. Times encoding of about 2 MB of text.
# demo.py - count tokens with tiktoken before you call the LLM
import os, time, json, tempfile
# Keep the encoding files in a folder we control (default is <tmp>/data-gym-cache)
CACHE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "tiktoken-cache")
os.environ["TIKTOKEN_CACHE_DIR"] = CACHE
import tiktoken
print("tiktoken", tiktoken.__version__, "| cache dir:", CACHE)
print("cached files before loading:", len(os.listdir(CACHE)) if os.path.isdir(CACHE) else 0)
# 1. Pick the encoding. First load downloads the BPE file; later loads read the cache.
for name in ("o200k_base", "cl100k_base"):
    t = time.perf_counter(); enc = tiktoken.get_encoding(name)
    print(f"load {name}: {(time.perf_counter()-t)*1000:.0f} ms, vocab size n_vocab={enc.n_vocab}")
o200k = tiktoken.get_encoding("o200k_base")
cl100k = tiktoken.get_encoding("cl100k_base")
print("\nencoding_for_model():")
for m in ("gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-5", "gpt-4", "gpt-3.5-turbo", "text-embedding-3-small"):
    print(f"{m:<24} -> {tiktoken.encoding_for_model(m).name}")
for m in ("claude-sonnet-4", "llama-3.1-8b"):
    try:
        tiktoken.encoding_for_model(m)
    except KeyError as e:
        print(f"{m:<24} -> KeyError: {str(e)[:70]}...")
# 2. Count tokens for different kinds of text
samples = {
    "English": "Please summarize this support ticket in three bullet points.",
    "Hindi (same meaning)": "कृपया इस सपोर्ट टिकट का तीन बिंदुओं में सारांश दें।",
    "Python": "def add(a: int, b: int) -> int:\nreturn a + b\n\nprint(add(2, 3))\n",
    "JSON": json.dumps({"user_id": 4821, "plan": "annual", "seats": 12, "tags": ["billing", "refund"]}),
}
print(f"\n{'sample':<22}{'chars':>6}{'utf8 bytes':>12}{'o200k':>7}{'cl100k':>8}{'chars/tok (o200k)':>19}")
for label, text in samples.items():
    a, b = len(o200k.encode(text)), len(cl100k.encode(text))
    print(f"{label:<22}{len(text):>6}{len(text.encode()):>12}{a:>7}{b:>8}{len(text)/a:>19.2f}")
# 3. Round-trip and token pieces
text = samples["English"]
ids = o200k.encode(text)
print("\nround-trip ok:", o200k.decode(ids) == text)
print("first 8 ids:", ids[:8])
print("first 8 pieces:", [o200k.decode_single_token_bytes(i) for i in ids[:8]])
hi = o200k.encode(samples["Hindi (same meaning)"])
print("Hindi first 6 pieces (bytes):", [o200k.decode_single_token_bytes(i) for i in hi[:6]])
print("decode of a single Hindi token:", repr(o200k.decode(hi[1:2])))
# 4. Special tokens: encode() refuses them unless you opt in
try:
    o200k.encode("user text with <|endoftext|> inside")
except ValueError as e:
    print("\nencode() with special token ->", type(e).__name__, str(e)[:60] + "...")
print("encode_ordinary() ->", len(o200k.encode_ordinary("user text with <|endoftext|> inside")), "tokens (treated as plain text)")
# 5. Truncate and chunk by tokens, not characters
def truncate_to_tokens(text, n, enc=o200k):
    ids = enc.encode(text)
    return text if len(ids) <= n else enc.decode(ids[:n])
def chunk_by_tokens(text, size, overlap=0, enc=o200k):
    ids = enc.encode(text)
    step = size - overlap
    return [enc.decode(ids[i:i+size]) for i in range(0, max(len(ids)-overlap, 1), step)]
doc = " ".join([samples["English"], samples["Hindi (same meaning)"], samples["Python"], samples["JSON"]]) * 3
print(f"\ndoc: {len(doc)} chars, {len(o200k.encode(doc))} o200k tokens")
short = truncate_to_tokens(doc, 20)
print("truncate_to_tokens(doc, 20) ->", len(o200k.encode(short)), "tokens:", repr(short))
chunks = chunk_by_tokens(doc, 64, overlap=8)
print("chunk_by_tokens(doc, 64, overlap=8) ->", len(chunks), "chunks, token sizes:", [len(o200k.encode(c)) for c in chunks])
by_chars = [doc[i:i+256] for i in range(0, len(doc), 256)]
print("same doc cut every 256 chars -> token sizes:", [len(o200k.encode(c)) for c in by_chars])
# 6. Budget the context window and estimate cost (formula only, EXAMPLE price)
def chat_tokens(messages, enc=o200k, per_message=3, reply_priming=3):
    # Cookbook heuristic for gpt-4o-style chat: ~3 tokens/message + 3 for the reply. An estimate.
    return sum(per_message + sum(len(enc.encode(v)) for v in m.values()) for m in messages) + reply_priming
messages = [
    {"role": "system", "content": "You are a concise support assistant."},
    {"role": "user", "content": samples["English"] + "\n\n" + doc},
]
raw = sum(len(o200k.encode(m["content"])) for m in messages)
est = chat_tokens(messages)
print(f"\nchat prompt: content tokens={raw}, with message overhead estimate={est} (+{est-raw})")
CONTEXT, MAX_OUTPUT = 8_000, 1_000 # example budget numbers for the demo
print(f"fits in a {CONTEXT}-token window with {MAX_OUTPUT} reserved for output:", est + MAX_OUTPUT <= CONTEXT)
EXAMPLE_PRICE_PER_MTOK = 1.00 # EXAMPLE ONLY, not a real quoted price
print(f"cost = tokens/1e6 * price -> {est}/1e6 * ${EXAMPLE_PRICE_PER_MTOK:.2f} = ${est/1e6*EXAMPLE_PRICE_PER_MTOK:.6f} per call (example price)")
# 7. Speed: encode a long text
long_text = (doc + "\n") * 2000
for name, enc in (("o200k_base", o200k), ("cl100k_base", cl100k)):
    enc.encode(long_text[:1000]) # warm-up
    times = []
    for _ in range(5):
        t = time.perf_counter(); n = len(enc.encode(long_text)); times.append(time.perf_counter() - t)
    times.sort()
    mb = len(long_text.encode()) / 1e6
    print(f"encode {mb:.2f} MB ({len(long_text):,} chars) with {name}: {n:,} tokens, median {times[2]*1000:.0f} ms ({mb/times[2]:.1f} MB/s)")
lines = long_text.splitlines()
t = time.perf_counter(); o200k.encode_batch(lines, num_threads=8); print(f"encode_batch over {len(lines):,} lines (8 threads): {(time.perf_counter()-t)*1000:.0f} ms")

3. Real output

This is from my second run, with the encoding files already cached. The first run's differences are covered in the gotchas below.

tiktoken 0.14.0 | cache dir: /workspace/thursday-2026-11-19-pm/tiktoken-cache
cached files before loading: 2
load o200k_base: 199 ms, vocab size n_vocab=200019
load cl100k_base: 80 ms, vocab size n_vocab=100277
encoding_for_model():
gpt-4o -> o200k_base
gpt-4o-mini-> o200k_base
gpt-4.1-> o200k_base
gpt-5-> o200k_base
gpt-4-> cl100k_base
gpt-3.5-turbo-> cl100k_base
text-embedding-3-small -> cl100k_base
claude-sonnet-4-> KeyError: 'Could not automatically map claude-sonnet-4 to a tokeniser. Please us...
llama-3.1-8b -> KeyError: 'Could not automatically map llama-3.1-8b to a tokeniser. Please use `...
sample charsutf8 byteso200kcl100kchars/tok (o200k)
English 6060 1010 6.00
Hindi (same meaning)51 135 1854 2.83
Python6767 2727 2.48
JSON7979 3130 2.55
round-trip ok: True
first 8 ids: [8256, 104833, 495, 2498, 16681, 306, 3407, 31926]
first 8 pieces: [b'Please', b' summarize', b' this', b' support', b' ticket', b' in', b' three', b' bullet']
Hindi first 6 pieces (bytes): [b'\xe0\xa4\x95', b'\xe0\xa5\x83', b'\xe0\xa4\xaa', b'\xe0\xa4\xaf\xe0\xa4\xbe', b' \xe0\xa4\x87\xe0\xa4\xb8', b' \xe0\xa4\xb8\xe0\xa4\xaa']
decode of a single Hindi token: 'ृ'
encode() with special token -> ValueError Encountered text corresponding to disallowed special token '...
encode_ordinary() -> 11 tokens (treated as plain text)
doc: 780 chars, 255 o200k tokens
truncate_to_tokens(doc, 20) -> 20 tokens: 'Please summarize this support ticket in three bullet points. कृपया इस सपोर्ट टिकट का तीन ब'
chunk_by_tokens(doc, 64, overlap=8) -> 5 chunks, token sizes: [64, 64, 64, 64, 31]
same doc cut every 256 chars -> token sizes: [84, 86, 85, 5]
chat prompt: content tokens=272, with message overhead estimate=283 (+11)
fits in a 8000-token window with 1000 reserved for output: True
cost = tokens/1e6 * price -> 283/1e6 * $1.00 = $0.000283 per call (example price)
encode 2.07 MB (1,562,000 chars) with o200k_base: 508,000 tokens, median 87 ms (23.7 MB/s)
encode 2.07 MB (1,562,000 chars) with cl100k_base: 726,000 tokens, median 153 ms (13.5 MB/s)
encode_batch over 26,000 lines (8 threads): 679 ms

How to read it

Same meaning, very different counts. The English sentence is 10 tokens under both encodings. The Hindi translation is 18 tokens with o200k_base and 54 with cl100k_base, so the same request costs 1.8x the English count on a gpt-4o-class model and 5.4x on a cl100k model. Devanagari takes 3 UTF-8 bytes per character (135 bytes for 51 characters), and the older vocabulary has far fewer merges for it.

Code and JSON are dense. The Python snippet came out at 2.48 characters per token and the JSON at 2.55, against 6.00 for the English sentence. Indentation, brackets, quotes and short identifiers all cost tokens. If your prompt is mostly tool output or JSON, a chars/4 estimate will undercount.

Round-trip is exact. decode(encode(text)) == text returned True. The token pieces show how BPE splits text. Common English words are single tokens with the leading space attached (b' summarize'). The Hindi tokens are much smaller, and one of them decoded to just the vowel sign 'ृ'.

Chunking by tokens gives predictable sizes. chunk_by_tokens(doc, 64, overlap=8) gave chunks of exactly 64 tokens (plus a 31-token tail). Cutting the same text every 256 characters gave chunks of 84, 86 and 85 tokens. On text with more code or non-English content, that drift gets bigger.

Message overhead is small but real. The two message contents were 272 tokens. With the Cookbook's heuristic for gpt-4o-style chat (3 tokens per message, plus 3 to prime the reply) the estimate is 283.

It's fast. Encoding 2.07 MB in one call took a median 87 ms with o200k_base (508,000 tokens) and 153 ms with cl100k_base (726,000 tokens). The same text took 30% fewer tokens with o200k because the text mixes in Hindi. Note that the long text is the same 780-character block repeated 2,000 times, so treat the speed as a rough figure.

How to budget a request

The check before every call is simple: prompt_tokens (incl. message overhead) + max_output_tokens <= model context window

Count the prompt with the encoding for your model (encoding_for_model(model)). Add a margin for chat formatting. The Cookbook says to treat its message formula as "an estimate, not a timeless guarantee." Tool/function definitions add extra tokens on top.

Reserve room for the output (max_output_tokens). The demo uses an example 8,000-token window with 1,000 tokens reserved for output. If it doesn't fit, shrink the biggest variable part first (retrieved chunks, chat history) with truncate_to_tokens or fewer chunks. Don't cut the system prompt.

Cost uses the same count: cost = input_tokens / 1e6 * input_price_per_MTok + output_tokens / 1e6 * output_price_per_MTok

In the demo I used an example price of $1.00 per million input tokens. That is not a quoted price. Look up your model on OpenAI's pricing page and plug in the current numbers. With that example price, a 283-token prompt costs $0.000283. The useful part is the ratio: if Hindi users send 1.8x the tokens of English users for the same request, your per-user cost is 1.8x too.

Gotchas I hit while testing

The first call downloads the encoding file. With an empty cache, loading o200k_base took 6,877 ms and cl100k_base 4,711 ms, because tiktoken fetched the BPE files over the network. Once cached, loading took 199 ms and 80 ms. By default the cache lives in your temp directory (<tmp>/data-gym-cache), which containers and serverless functions often wipe. Set

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.