How AI Text Detectors Actually Work (And Why They Flag You)

Every few months someone forwards me a screenshot: a detector says their essay is "98% AI-generated," and they wrote every word themselves. Usually they're a non-native English speaker. I've spent enough time inside these systems to explain why that happens, and why the number on the dial is not a probability in any sense you'd want to bet on.

Blunt version up front: statistical AI-text detection is a real technique with a real signal, and it is also fundamentally incapable of the thing people use it for - adjudicating individual documents.

What the classifiers are actually measuring

Almost every zero-shot detector reduces to one idea: run the candidate text through a language model and ask how surprised the model is. The core quantity is per-token log-likelihood:

import math

def mean_logprob(tokens, model):
    """Average log P(token | prefix). Higher = text is predictable."""
    total = 0.0
    for i, tok in enumerate(tokens):
        dist = model.next_token_distribution(tokens[:i])
        total += math.log(dist[tok] + 1e-12)
    return total / len(tokens)

def perplexity(tokens, model):
    return math.exp(-mean_logprob(tokens, model))

Low perplexity means the scoring model found the text easy to predict. Since decoding strategies used in practice - greedy, low-temperature sampling, nucleus sampling with modest top-p - prefer high-probability continuations, generated text tends to sit in a lower-perplexity region than unconstrained human writing. That's the entire foundation.

"Burstiness" is the second-moment version of the same observation. Human writing varies: a dense clause, then a short one, a surprising word choice, then three predictable ones. So the variance of per-sentence perplexity tends to run higher for humans. Detectors feed both mean and variance into a threshold or small classifier.

The more interesting modern approach is curvature-based - DetectGPT and its faster descendants. The insight: machine text tends to sit near a local maximum of the model's log-probability surface. Perturb it slightly (mask-and-refill spans), rescore, measure the drop:

def curvature_score(text, model, perturb, k=20):
    base = mean_logprob(tokenize(text), model)
    drops = [base - mean_logprob(tokenize(perturb(text)), model) for _ in range(k)]
    return sum(drops) / len(drops)  # large positive => near a local max

Human text, already off-peak, doesn't drop much when you jiggle it. A cleverer signal than raw perplexity - and it still inherits the flaw below.

The flaw: you are measuring fluency-typicality, not authorship

Every one of these scores is a proxy. What they measure is how closely this text tracks the scoring model's expectations. Authorship is not in the equation anywhere. It's inferred, on the assumption that "typical for the model" correlates with "produced by a model." That assumption breaks in predictable ways:

• Non-native writers. Second-language writing often uses a smaller, more conventional lexicon and more regular syntax - precisely because the writer deploys reliably-learned constructions rather than idiomatic risks. That is low perplexity by construction. This isn't a hypothesis; a well-known 2023 Stanford study by Liang et al. found GPT detectors flagged TOEFL essays by non-native English writers as AI-generated at dramatically higher rates than native-writer essays, which were classified near-correctly. The detectors were, in effect, measuring English fluency and reporting it as machine authorship.

• Formulaic genres. Lab reports, legal boilerplate, clinical notes, and technical documentation are low-entropy by design - the point is conventional phrasing. A human-written incident postmortem can score more "AI-like" than a chatty blog post that was actually generated.

• Domain mismatch. The score depends on the scorer's training distribution. Score a language or register it under-represents and perplexity rises for reasons unrelated to who wrote it - which is why accuracy claims measured on English news never transfer to Japanese academic prose.

• Editing. Light human revision measurably moves perplexity and curvature scores - the signal degrades under exactly the workflow most common in practice.

Why the "99% accurate" number is misleading

Even granting a good classifier, base rates destroy individual-document use. Suppose a detector has 95% sensitivity and 95% specificity - better than most honest evaluations - and 5% of submissions are machine-written. Out of 10,000 documents:

• truly AI (500): 475 flagged, 25 missed
• truly human (9500): 475 flagged (false positives!), 9025 clear

Of 950 flagged documents, 475 are innocent. Precision = 50%. A coin flip, from a classifier you'd call excellent. Push specificity to 99% and precision only reaches about 84% - still one wrongly-accused person in six.

And the false positives aren't randomly distributed: they concentrate on non-native writers and formulaic genres, so the harm lands unevenly on the people least equipped to contest it.

This is standard screening-test arithmetic, and it's why honest detector documentation says "not for disciplinary decisions." The problem is not calibration. It's that a probabilistic signal cannot become a verdict just because a UI renders it as a big percentage.

How to use these tools without lying to yourself

The signals are useful - just not as verdicts:

• Aggregate, don't adjudicate. Detector scores over a corpus of 50,000 documents tell you something real about distribution shift. On one document they tell you almost nothing.
• Report a distribution, not a badge. Show the score against a reference distribution for that genre and language. "Perplexity 21, the 12th percentile for technical documentation" is honest. "98% AI" is not.
• Calibrate per language and register. A threshold tuned on English blog posts is meaningless on Korean academic writing. Morphologically rich languages tokenize differently and land in a different perplexity range before you've measured anything about the author.
• Never let a score be sole evidence. Version history, drafts, and a five-minute conversation about the content are all higher-signal than any classifier.

When I was building a per-language calibration harness, the multilingual tooling at Suikou AI was a useful comparison point precisely because it treats detection and rewriting as two views on one distribution problem - and because the Japanese and Korean side makes the tokenization issue impossible to ignore: change your morphological segmentation and every perplexity number moves, with no change to the text at all.

That's the whole lesson. If a metric shifts when you change your tokenizer, it was never measuring authorship.

Top comments (0)