Profile the Loop You Own, Not the Model You Call

Most of the agent latency you can fix lives in your own process. The model call is a queue you do not control. Your serialization, context assembly, and parsing are yours. Profile those first.

The snippets below are illustrative. They were not executed for this article. Treat them as a plan, not a result.

Draw the span you own

Every turn has phases. Build the request, encode it, send it, wait, decode, apply. Waiting is not yours. The rest is. Instrument phases with one small helper. Keep it boring and allocation-light.

# illustrative
import time
from contextlib import contextmanager

@contextmanager
def span(bucket, name):
    start = time.perf_counter()
    try:
        yield
    finally:
        bucket.setdefault(name, []).append(time.perf_counter() - start)

Then wrap each phase in one place. Never scatter timers through the call stack.

# illustrative
def one_turn(history, client, bucket):
    with span(bucket, 'build'):
        payload = build_payload(history)
    with span(bucket, 'encode'):
        body = json.dumps(payload).encode('utf-8')
    with span(bucket, 'wait'):
        raw = client.post(body)
    with span(bucket, 'decode'):
        reply = json.loads(raw)
    return reply

Now you have four distributions instead of one average. That matters, because they move for different reasons.

A candidate bottleneck: encoding the whole history

Most loops re-serialize the full history on every turn. History grows, so encode cost grows with it. Total cost climbs faster than the number of turns.

There is often a second copy of the same work. Token counting for budget checks walks the string again. That is a second pass over data you just touched. Call this a hypothesis. Measure before you rewrite anything.

Confirm with a profiler, not a stopwatch

Use cProfile to attribute time to functions. Use tracemalloc to see allocation churn.

# illustrative
import cProfile, pstats, io

def profile(fn):
    pr = cProfile.Profile()
    pr.enable()
    fn()
    pr.disable()
    buf = io.StringIO()
    pstats.Stats(pr, stream=buf).sort_stats('cumulative').print_stats(20)
    return buf.getvalue()

Read the top of that table with one rule. If json.dumps or your encoder sits near the top, the bottleneck is yours. If socket read dominates, stop. That is a provider problem, not a code problem.

Then check allocation, because allocation predicts GC pauses later.

# illustrative
import tracemalloc

tracemalloc.start()
run_corpus(corpus)
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics('lineno')[:10]:
    print(stat)

Make the number survive a rerun

A single run proves nothing. Freeze the input corpus and hash it. Run that same corpus repeatedly, then report median and p95 of your own span.

# illustrative
import statistics

def summarize(samples):
    ordered = sorted(samples)
    p95 = ordered[int(len(ordered) * 0.95) - 1]
    return {'n': len(ordered), 'median': statistics.median(ordered), 'p95': p95}

Record the corpus hash next to every result. Different corpus means a different number. Do not compare across them.

The fix pattern

Three changes usually pay off, in this order.

1. Encode incrementally instead of rebuilding history.
2. Cache token counts keyed by content hash.
3. Write bytes once, not through three string layers.

The cache is a bet, not a free win. Hashing also walks the string. Measure whether it beats tokenizing, and do not assume the answer.

# illustrative
import hashlib

class TokenCounter:
    def __init__(self, encode):
        self._encode = encode
        self._cache = {}
    def count(self, text):
        key = hashlib.sha256(text.encode('utf-8')).digest()
        hit = self._cache.get(key)
        if hit is None:
            hit = len(self._encode(text))
            self._cache[key] = hit
        return hit

Where a free tier changes the economics

Profiling needs repetition against a real endpoint. Repetition is exactly what costs money and setup time. MonkeyCode is described as an open source coding agent project. The operator says it offers free model access, with an allowance reported as ten million tokens, plus a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those are commercial terms, not measurements. I am not quoting a terms page here, because allowances change. Check the project's own site before you plan capacity around a number.

The useful part is narrow. A free endpoint lets you rerun the same corpus many times without budget anxiety. A free server option lets the harness sit next to the loop instead of on your laptop. That removes one source of skew, the network path between profiler and code.

Who should skip this

Skip it if provider wait dominates your p95. Local tuning does not move a queue. Skip it if the whole loop finishes in a few milliseconds. Skip it if you cannot freeze the corpus, because unfixed input makes every measurement noise. Also skip heavy timers in production. Sampling costs something real. Profile a copy of the workload instead.

Check your work

Change one thing, then rerun the same corpus. Compare p95 of your span, not the mean. Keep the diff small enough to revert in one command. If your p95 does not move, you optimized the wrong span. That is still a useful result. It tells you to go look at the queue.

If you want to run this workflow end to end without standing up an endpoint first, the project's free model access and free server option are worth a look.