Is That API Really Serving the Model It Claims? A Reproducible Fingerprint Test

The idea: compare distributions, not prose

The paper One Token Is Enough shows that models have surprisingly stable preferences when they answer simple one-token questions repeatedly. Ask for a random number, color, letter, city, or coin flip 25 times. One answer means almost nothing. The distribution across many answers is a behavioral fingerprint.

A practical audit looks like this:

• Select 8 or more task-language cells.
• Sample each cell repeatedly at a fixed temperature.
• Normalize the one-word answers.
• Compare the observed distributions with a trusted reference using Jensen-Shannon divergence.
• Repeat the run on another day before drawing a strong conclusion.

The paper reports an equal error rate of about 10.6% with 8 cells and about 7.3% with 40 cells. That is useful evidence, not perfect identification.

Four verdicts, not a binary accusation

The open-source implementation we are using reports four states:

• match: behavior is statistically consistent with the reference.
• uncertain: the result is in the gray zone; collect more samples or repeat later.
• mismatch: behavior differs substantially; investigate before concluding why.
• insufficient: there are not enough valid comparable samples.

A mismatch does not prove fraud. Model updates, quantization, hidden system prompts, reasoning fallbacks, and load-balancing across several backends can all move the fingerprint. That limitation is part of the result, not fine print.

Run the test locally

The MIT-licensed llm-fingerprint-detector provides a TypeScript CLI and library for OpenAI-compatible endpoints. The fixed experiment definitions, public report schema, and blank result card are in the public protocol directory.

Keep the API key in a local environment variable. Do not paste it into a form, chat, issue, or public log.

git clone https://github.com/ToseaAI/llm-fingerprint-detector.git
cd llm-fingerprint-detector
npm install
npm run build
export LLM_FINGERPRINT_API_KEY='your-local-key'
node dist/cli.js fingerprint \
  --base-url https://your-endpoint.example.com/v1 \
  --model your-model-id \
  --out observed.json

To verify against a trusted reference:

node dist/cli.js verify \
  --base-url https://your-endpoint.example.com/v1 \
  --model your-model-id \
  --reference trusted-reference.json

Record the model ID, test-set version, timestamp, temperature, sample count, comparable cells, mean JSD, split-half consistency, warnings, and a hash of the response summary. Never publish the key or private prompts.

The test I want providers to publish

A provider should not publish only a green badge. A useful public report should include:

We will run the same test on AllRouter at a fixed time for seven days and publish match, uncertain, mismatch, or insufficient without forcing a positive conclusion.

Why this can be more useful than a generic benchmark

Benchmarks ask whether a model is capable. A routing audit asks whether the endpoint stays consistent with the model and configuration you chose. That matters for coding agents. A silent route change can alter formatting discipline, tool-use behavior, latency, refusal boundaries, and the amount of review a patch needs. Even when a response is syntactically valid, the workflow cost can change.

The identity test should therefore sit next to a small task suite:

• preserve a public API during a refactor;
• handle missing configuration explicitly;
• follow a strict JSON schema;
• separate risk analysis from the edit;
• keep the diff inside the requested boundary.

The fingerprint tells you whether behavior changed. The task suite tells you whether that change matters to your workflow.

Re-run the same experiment on AllRouter

If you want to compare the same OpenAI-compatible workflow on AllRouter, use the campaign-specific entry below:

Run the same test on AllRouter.

The goal is not “trust this router.” The goal is the opposite: make the result reproducible enough that trust is not the only evidence available. In the next report, we will publish the seven-day drift format, including latency, error rate, split-half consistency, and the exact decision thresholds.