"How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method"

How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method Detect whether an LLM was trained from scratch or derived from Qwen, Llama, or DeepSeek - by fingerprinting architecture, tokenizer, and weight provenance from public Hugging Face artifacts. Includes the two traps almost everyone hits. Keywords: LLM provenance · model fingerprinting · from-scratch vs fine-tuned · architecture signature · tokenizer overlap · embedding CKA · model lineage · Korean sovereign AI · open-weight derivatives When a lab announces a "from-scratch, self-developed" foundation model, can an outsider verify that claim - using nothing but public files? In late July 2026, several Korean labs shipped DeepSeek-rivaling "self-developed" models (e.g. LG's 750B K-EXAONE 2.0). The claim triggered a debate that spilled well beyond Korea: a single Zhihu thread crossed 2.7 million views, asking whether these models were trained from scratch or quietly built on Qwen / Llama / DeepSeek. That question is answerable - quantitatively, reproducibly, from public artifacts. This post is the method. Everything below runs against any two repos on the Hugging Face Hub, and there's a live tool at the end. Framing up front: building on open-weight bases (Qwen, Llama, DeepSeek, Mistral) is a legitimate, industry-standard practice. This is about provenance transparency, not accusation. The same yardstick applies to every model. The idea: three independent fingerprints A model leaves three separable fingerprints in its public files: - Architecture - the shape declared in config.json - Tokenizer - the vocabulary in tokenizer.json - Weights - the learned representation in model.safetensors Each answers a different question, and - crucially - they can disagree. That disagreement is where the signal lives. Fingerprint 1 - Architecture (config.json ) Every transformers checkpoint ships a config.json . Six fields form a surprisingly discriminative signature: import requests FIELDS = ("model_type", "vocab_size", "hidden_size", "intermediate_size", "num_hidden_layers", "num_attention_heads", "num_key_value_heads") def arch_fingerprint(repo: str) -> dict: url = f"https://huggingface.co/{repo}/resolve/main/config.json" c = requests.get(url, headers={"User-Agent": "genome/1.0"}, timeout=30).json() return {k: c.get(k) for k in FIELDS} The shape tuple (hidden_size, intermediate_size, num_hidden_layers, heads, kv) is effectively a fingerprint of the reference architecture. One matching field is a coincidence; five matching simultaneously is not. A few real matches I measured: | Shape (hidden · inter · layers · heads · kv) | Exact match | |---|---| | 3584 · 18944 · 28 · 28 · 4 | Qwen2.5-7B | | 8192 · 29568 · 80 · 64 · 8 | Qwen2.5-72B | | 5120 · 17408 · 40 · 40 · 8 | Qwen3-14B | | 4096 · 14336 · 32 · 32 · 8 | Llama-3.1-8B | | 7168 · 18432 · 61 · (moe 2048) | DeepSeek-V3 | An exact tuple match is strong evidence the architecture was adopted, not independently designed. Fingerprint 2 - Tokenizer (a paternity test) Architecture alone can mislead: a model can adopt a foreign architecture but train a genuinely new tokenizer (or vice-versa). Measure the tokenizer directly, comparing vocabularies with a min-overlap ratio: def vocab_set(repo: str) -> set: url = f"https://huggingface.co/{repo}/resolve/main/tokenizer.json" v = requests.get(url, timeout=60).json()["model"]["vocab"] # BPE: {token: id} return set(v.keys()) def tokenizer_overlap(a: str, b: str) -> float: A, B = vocab_set(a), vocab_set(b) return len(A & B) / min(len(A), len(B)) # 1.0 == one is a subset of the other This surfaces what config hides. One model matched Qwen2.5-7B's architecture exactly, yet its tokenizer overlapped Qwen by only ~0.38 - a "foreign brain, own language" case: adopted architecture, freshly trained (Korean) tokenizer. Others reused a base tokenizer verbatim (overlap = 1.000), confirming a straight fine-tune. Why min , not union? Usingmin(|A|, |B|) in the denominator makes a reduced vocabulary that is a strict subset of a larger one score ~1.0 - the correct signal for "carved out of the base." A Jaccard (union) denominator would wrongly dilute that. Fingerprint 3 - Weights (here be dragons) The gold-standard question: were the weights trained from scratch, or continued-pretrained on a foreign base? Load the token embeddings and compare. Two traps await. First, a helper to pull only the embedding tensor (no need to download the whole model): import json, torch from huggingface_hub import hf_hub_download from safetensors import safe_open def load_embedding(repo: str) -> torch.Tensor: try: idx = hf_hub_download(repo, "model.safetensors.index.json") shard = json.load(open(idx))["weight_map"]["model.embed_tokens.weight"] except Exception: shard = "model.safetensors" path = hf_hub_download(repo, shard) with safe_open(path, framework="pt") as f: key = next(k for k in f.keys() if k.endswith("embed_tokens.weight")) return f.get_tensor(key).float() Trap 1 - row-wise cosine is useless The naive approach: for shared tokens, average the row-wise cosine similarity of the two embedding matrices. Shared lineage → similar embeddings, right? Wrong - even when lineage is obvious. I measured near-zero mean cosine for both a known from-scratch model and a known Llama-derivative. The culprit is rotational invariance: a Transformer's hidden space has no privileged basis, so two models can encode identical information under an arbitrary orthogonal rotation. Row-wise cosine reads rotation as dissimilarity and tells you nothing about lineage. Trap 2 - CKA helps, but is not conclusive Linear CKA (Centered Kernel Alignment) is invariant to rotation and isotropic scaling - the right tool for comparing representations: def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float: # X: (n, d1), Y: (n, d2) - SAME token order (shared vocabulary) X = X - X.mean(0, keepdim=True) Y = Y - Y.mean(0, keepdim=True) num = (X.T @ Y).norm() ** 2 den = (X.T @ X).norm() * (Y.T @ Y).norm() return (num / den).item() A from-scratch model scored near-zero CKA against its candidate base - clean evidence of independent pretraining. But a continued-pretrained derivative scored only ~0.25 - barely above the ~0.21 baseline between two unrelated models of the same family. Large-scale training reshapes embeddings enough that CKA loses discriminative power on the derivative side. The honest conclusion: the weights axis reliably confirms from-scratch (near-zero CKA), but it is not a strong detector of derivation. For that, architecture + tokenizer fingerprints stay primary. Report the weights axis as supporting evidence, never as a standalone verdict. (This is the single most important caveat in the whole method - and the one most write-ups omit.) Bonus fingerprint - attention diversity as an originality proxy Most models declare one attention mechanism; a few mix several. The count of distinct mechanisms in config.json is a cheap proxy for architectural originality: KEYS = ("layer_types", "linear_attn_config", "sliding_window", "mamba2_d_state", "hyena_filter_order", "mla_kv_lora_rank", "attention_cls") def attention_diversity(cfg: dict) -> list: # e.g. layer_types = [full_attention×16, sliding_attention×48] -> hybrid (2 kinds) return [k for k in KEYS if k in cfg] In my sweep, most models used a single grouped-query or multi-head-latent attention; some used a hybrid (layer_types=[full×16, sliding×48] ); the most diverse combined mamba2, hyena, MLA, linear attention, gated-delta-net, native-sparse-attention, and sliding-window in one stack. Combining axes → a single genotype Collapse the two primary axes (architecture × weights) into one label: | Genotype | Architecture | Weights | |---|---|---| | 🟢 Native | self | from-scratch | | 🔵 Adapted | mostly self | one axis borrowed | | 🟡 Mixed | partial | partial inheritance | | 🔴 Ported | foreign (exact match) | inherited | Keep tokenizer overlap and attention diversity beside the verdict, not folded into it, so readers can audit the raw evidence. Results: nine organizations, one yardstick Applying the identical pipeline to nine organizations' public foundation models (spanning large enterprises, telcos, mid-size firms, and startups), the picture is not uniform: some models match a foreign architecture and tokenizer exactly (Ported); others are self-built with no foreign match (Native); many sit in between. The per-model breakdown - 3D lineage graph, search, EN/中文/한국어, light + dark mode - is in the interactive tool below. Reproduce it yourself The functions above are the method. Point them at any two Hub repos: print(arch_fingerprint("some/model")) print(tokenizer_overlap("some/model", "Qwen/Qwen3-14B")) # weights (shared-vocab pair): X = load_embedding("candidate/model") Y = load_embedding("Qwen/Qwen3-1.7B") n = min(len(X), len(Y)) print("CKA:", linear_cka(X[:n], Y[:n])) # near-zero => from-scratch Limitations & honesty - Not an accusation. Open-weight reuse is legitimate and widespread. This reports lineage, not wrongdoing. - Weights axis is supporting, not conclusive (Trap 2). - Same yardstick for every model, without exception. - All inputs are public; corrections are welcome. Frequently asked questions How can you tell if an LLM was trained from scratch or fine-tuned from another model? Compare its config.json shape signature (hidden size, intermediate size, layer count) and its tokenizer.json vocabulary against known open-weight bases. An exact architecture match plus high tokenizer overlap indicates a derivative; a self-designed architecture with near-zero embedding CKA against candidate bases indicates from-scratch training. What is CKA (Centered Kernel Alignment), and why use it instead of cosine similarity? CKA is a rotation- and isotropic-scale-invariant similarity measure for neural-network representations. A Transformer's hidden space has no privileged basis, so plain row-wise cosine similarity is fooled by arbitrary orthogonal rotations between two models. CKA is not