AI citation checks: why a working link can't prove a quote

AI citation checks have to test the quoted words, because a link that opens only shows that a page exists at that address.

The check that works is cheap: remove copy noise, require the quote to appear as one exact run of whole tokens in a stored copy of the source, and use similarity scores only to show the reader the closest passage.

What our lab found

Local simulation with documents and bad quotes we wrote. It tests the checker, not any AI model or product.

The lab (labs/ai-10, Node.js 22, seed 42) writes ten short documents of our own and builds 60 answers, each citing one local file with one quote. Thirty quotes are copied from their file, a third of them with copy noise such as a curly apostrophe or a line break. The other 30 were built by hand: 10 real sentences cited to the wrong file, 10 merged from two half sentences, and 10 rewordings (3 of them with one changed number).

Citation links that open a real file: 60/60. All 60 cited local files existed and were readable, for the good answers and the bad ones. The lab makes no HTTP requests; in production the same blind spot belongs to an HTTP status check. The link opened; the slot where the quote should be is empty.

Shape / What happened / Exact check / Overlap score in lab:

• Wrong document - A real sentence, cited to the wrong file - Caught 10/10 - 0.10 to 0.33
• Merged - Two half sentences from one file joined - Caught 10/10 - 0.50 to 0.73
• Reworded - 1 to 3 words swapped, or one number changed - Caught 10/10 - 0.79 to 0.95

Rewordings are the hard case, because they share most of their tokens with a real sentence.

How do AI citation checks work?

Normalize both texts, then search for the quote as one contiguous run. Raw byte equality rejected 10 of 30 real quotes in the lab; answer A19 has two spaces after "When" where the page has one.

• Grounded quotes verified by exact match (after normalization): 30/30
• Grounded quotes that a raw, un-normalized match would wrongly reject: 10/30

The lab's normalizer applies Unicode NFKC (UAX #15), lowercases, turns punctuation into spaces and collapses whitespace. Every step can also erase meaning:

• Step / Removes noise like / Can also erase
• NFKC / Full-width digits, ligatures, NBSP / Superscripts: "10²" becomes "102"
• Lowercase / Sentence-start capitals / Case-sensitive IDs: "US" and "us"
• Punctuation to space / Curly quotes, commas / Signs and decimal points: "-0.5" and "0.5" both become "0 5"
• Collapse whitespace / Line wraps, double spaces / Code indentation

So fold typography and keep numbers as written, with sign, decimal point, exponent, superscripts and percent sign attached. The verifier also has to look just outside the matched span: in "set it to - 0.5", the quote "0.5" is a whole token of the source, and the spaced minus in front still changes the value.

A number it cannot parse ("1/2", "10-20", "3:00") gets "could not check". The lab's normalizer does none of this; the verifier in Code does.

Normalization also moves positions (NFKC turns "fi" into two characters), so map each token to its original span and slice the original text. With N source tokens and M quote tokens, exact search by Knuth, Morris and Pratt is O(N + M). The closest-passage search for display is O((N - M + 1) · M): for N = 100,000 and M = 30, that is 99,971 × 30 = 2,999,130 comparisons per failed quote, so run it only inside the cited chunk.

The lab's median was 0.055 ms per answer, on warmed reads of tiny local files.

Why does a 0.9 similarity score accept a changed number?

The lab's fuzzy score slides a window the length of the quote over the source and takes the best fraction of quote tokens found in it, ignoring order: threshold grounded-rejected ungrounded-accepted paraphrase-accepted number-flips-accepted exact 0 0 0 0 0.95 0 1 1 0 0.90 6 6 3 0.80 9 9 3 0.70 11 10 3. At 0.9, 6 of 10 rewordings pass, all 3 number changes among them. The second column is zero on every row: looseness saved no real quote that normalization had not already saved.

One changed digit, 15 of 16 tokens still match. Answer A59 cites cdn.txt. The page says "the origin serves only 1 request in 10"; the quote says 2. Both sentences normalize to 16 tokens, and 15 match: score = matching tokens / quote tokens = 15 / 16 = 0.9375 (printed 0.94). 0.94 >= 0.90 -> accepted.

With k substitutions in an n-token quote, the aligned window keeps at least n - k matching tokens, so the score is at least (n - k) / n, and another window or a reordering can score higher. One substitution therefore passes at 0.9 whenever n ≥ 10. A changed number and a changed article both cost 1/n.

Citations as spans, snapshots and hashes

The strongest design copies the quote out of the source, so the model's wording never reaches the quotation marks. Cite spans. Retrieval already holds each chunk's text and offsets (how RAG retrieval finds one line in 100 pages). Ask the model for chunk IDs and render the quote from stored text, using the W3C Web Annotation TextQuoteSelector and TextPositionSelector.

Snapshot the original. Store the fetched bytes and the extracted text, each with a SHA-256, plus extractor and normalizer versions. Only a snapshot tells a changed page from an invented quote, the copy-versus-origin problem from CDN caching. Use conditional creation (S3 If-None-Match: *) to prevent accidental overwrites. For retained snapshots, configure Object Lock retention and store the object version ID. Verify the digest on every read; a hash detects change and cannot stop it.

Verify before display. If the model wrote the quote, check it against the snapshot, render the stored span on a pass, and drop the quotation marks on a fail. Text fragments help readers locate a passage; verify against the stored snapshot before display.

An internal citation record:

{ "citationId" : "c-7f3a" , "snapshotId" : "sha256:9b1e...c04d" , "fetchedAt" : "2026-11-05T14:02:11Z" , "url" : "https://example.com/cdn" , "extractor" : "html-text@3.2" , "normalizer" : "tok-nfkc-lower@2" , "hashes" : { "raw" : "sha256:9b1e...c04d" , "text" : "sha256:41aa...e9f2" , "norm" : "sha256:c3d0...7b18" } , "selector" : { "position" : { "unit" : "utf16" , "start" : 38 , "end" : 76 } , "quote" : { "exact" : "the origin serves only 1 request in 10" , "prefix" : "90 percent means " , "suffix" : "." } } , "matches" : 1 }

The offsets are JavaScript UTF-16 code units, and the unit is stored with them. A W3C TextPositionSelector counts Unicode code points, so convert UTF-16 offsets to code-point offsets when exporting one. The two counts differ after a supplementary-plane character.

What should the app show when a quote cannot be found?

• Situation / Show
• Exact match in the snapshot / The quote in quotation marks, linked to the passage, labelled "quote found"
• No match, a similar passage exists / No quotation marks; "Closest passage in the source:" with the differing words highlighted
• No match, nothing similar / "The cited source does not contain this quote." Keep the link, mark the claim unsupported
• Source fetch failed / "Could not check the source." Never a green tick

Use fuzzy matching to find the nearest passage for display, so the reader sees "1" next to "2". Label a pass "quote found": "source verified" claims more than the check proves.

Code

The verifier (labs/ai-10/quote-check.mjs) keeps each token's original span, keeps Devanagari marks and joiners, and compares numbers and operators as written, so "x = 5" does not match "x != 5". An unknown operator run such as "5!" gets "could not check", and a match is refused when a number at either end touches a sign, unit or operator just outside the span.

// quote-check.mjs·node quote-check.mjs
// Tokens keep their original span. Offsets are JavaScript UTF-16 code units.
const W = String . raw `\p{L}\p{M}\p{N}‌‍` ; // ZWNJ and ZWJ stay inside words
const NUM = String . raw `[+\-−]?(?:\p{Nd}+(?:[.,]\p{Nd}+)*|[.,]\p{Nd}+)` + String . raw `(?:[eE][+\-−]?\p{Nd}+)?[⁰¹²³⁴⁵⁶⁷⁸⁹⁺⁻]*[%‰]?` ;
// An operator run is kept whole and as written: "!=" is not "=". It is a token when it holds a
// comparison sign or sits next to a digit, so "real-time" stays prose.
const OP = String . raw `[!+\-−±/×÷*^=<>≤≥≠~≈]+` ;
const SYM = String . raw `[%‰°\p{Sc}]|(?<=\p{N}\s*)${OP}|${OP}(?=\s*[.,]?\p{N})|(?=[!+\-−±/×÷*^~]*[=<>≤≥≠≈])${OP}` ;
const KNOWN = new Set ( [ '+' , '-' , '−' , '±' , '/' , '×' , '÷' , '*' , '^' , '~' , '≈' , '=' , '<' , '>' , '≤' , '≥' , '≠' , '==' , '!=' , '<=' , '>=' , '->' , '=>' , '<-' ] ) ;
const TOKEN = new RegExp ( String . raw `(?<![${W}])(?<!\p{N}[^\s${W}])(?<num>${NUM})(?![${W}])(?![^\s${W}]\p{N})` + String . raw `|[${W}]+|(?<sym>${SYM})` , 'gu' ) ;
function tokenize ( text ) { const out = [ ] ; for ( const m of text . matchAll ( TOKEN ) ) { const raw = m [ 0 ] ; let t ; if ( m . groups . num ) t = 'n:' + raw ; // number: compared as written else if ( m . groups . sym ) // sign, unit or operator: a token; t = ( /^ [ % ‰°\ p { Sc } ] $ / u . test ( raw ) || KNOWN . has ( raw ) ? 's:' : 'bad:' ) + raw ; // "5!", "===": bad else if ( !/ \ p { N } / u . test ( raw ) ) t = 'w:' + raw . normalize ( 'NFKC' ) . toLowerCase ( ) ; else if ( / \ p { L } / u . test ( raw ) ) t = 'id:' + raw ; // IPv4, x², 5ms: exact else t = 'bad:' + raw ; // 1/2, 10-20, 3:00, ½ out . push ( { t , start : m . index , end : m . index + raw . length } ) ; } return out ; }
function findAll ( q , d ) { // KMP over token arrays: O(N + M) const fail = [ 0 ] ; for ( let i = 1 , k = 0 ; i < q . length ; i ++ ) { while ( k && q [ i ] !== q [ k ] ) k = fail [ k - 1 ] ; if ( q [ i ] === q [ k ] ) k ++ ; fail [ i ] = k ; } const hits = [ ] ; for ( let i = 0 , k = 0 ; i < d . length ; i ++ ) { while ( k && d [ i ] !== q [ k ] ) k = fail [ k - 1 ] ; if ( d [ i ] === q [ k ] ) k ++ ; if ( k === q . length ) { hits . push ( i - q . length + 1 ) ; k = fail [ k - 1 ] ; } } return hits ; }
function closest ( q , d ) { // display only: O((N - M + 1) * M) let best = { score : 0 , at : - 1 } ; for ( let s = 0 ; s + q . length <= d . length ; s ++ ) { let hit = 0 ; for ( let i = 0 ; i < q . length ; i ++ ) if ( d [ s + i ] === q [ i ] ) hit ++ ; if ( hit / q . length > best . score ) best = { score : hit / q . length , at : s } ; } return best ; }
// A number at either end of a match must not sit next to a sign, unit, operator
// or another number just outside the match, across spaces only: "- 0.5", "5 %", "1 / 2".
const touchy = ( x ) => x !== undefined && /^ ( n | s | bad ) : / . test ( x . t ) ;
const isNum = ( x ) => /^ ( n | id ) : / . test ( x . t ) ;
function contextOk ( source , dt , h , m ) { const first = dt [ h ] , last = dt [ h + m - 1 ] , prev = dt [ h - 1 ] , next = dt [ h + m ] ; const blank = ( a , b ) => /^ \ s * $ / u . test ( source . slice ( a , b ) ) ; if ( isNum ( first ) && touchy ( prev ) && blank ( prev . end , first . start ) ) return false ; if ( isNum ( last ) && touchy ( next ) && blank ( last . end , next . start ) ) return false ; return true ; }
export function checkQuote ( quote , source ) { const qt = tokenize ( quote ) , dt = tokenize ( source ) ; if ( qt . length === 0 ) return { found : false , reason : 'empty quote' } ; if ( qt . some ( ( x ) => x . t . startsWith ( 'bad:' ) ) ) return { found : false , reason : 'could not check: numeric form' } ; const q = qt . map ( ( x ) => x . t ) , d = dt . map ( ( x ) => x . t ) ; const hits = findAll ( q , d ) ; if ( hits . length === 0 ) { const c = closest ( q , d ) ; const text = c . at < 0 ? '' : source . slice ( dt [ c . at ] . start , dt [ c . at + q . length - 1 ] . end ) ; return { found : false , closest : { score : c . score , text } } ; } const ok = hits . filter ( ( h ) => contextOk ( source , dt , h , q . length ) ) ; if ( ok . length === 0 ) return { found : false , reason : 'could not check: numeric context' } ; const spans = ok . map ( ( h ) => ( { start : dt [ h ] . start , end : dt [ h + q . length - 1 ] . end } ) ) ; const { start , end } = spans [ 0 ] ; return { found : true , start , end , original : source . slice ( start , end ) , matches : spans . length } ; }
const page = 'A cache hit ratio of 90 percent means the origin serves only 1 request in 10.' ;
console . log ( checkQuote ( 'the origin serves only 1 request in 10' , page ) ) ;
console . log ( checkQuote ( 'the origin serves only 2 request in 10' , page ) ) ;
console . log ( checkQuote ( 'WhentheORIGIN serves only 1 request in 10' , page ) ) ;
console . log ( checkQuote ( 'x != 5' , 'stop if x != 5' ) . found ) ; // true: the operator matches as written
// Regression cases: every line must end in false.
const bad = [ [ '0.5' , 'set it to -0.5' ] , [ '0.5' , 'set it to - 0.5' ] , [ '2' , 'wait 20 seconds' ] , [ '1.000' , 'about 1,000 users' ] , [ '5' , 'set to .5' ] , [ '5' , 'a 5% loss' ] , [ '5' , 'a 5 % loss' ] , [ '10 2' , 'about 10² requests' ] , [ '1 2' , 'use 1/2 of it' ] , [ '1' , 'use 1 / 2 of it' ] , [ '!!!' , 'arbitrary source' ] , [ 'अनुपस्थित' , 'अर्को पाठ' ] , [ 'x = 5' , 'x != 5' ] , [ 'x = 5' , 'x ≠ 5' ] , [ 'a = b' , 'a != b' ] , [ 'x <= 5' , 'x < 5' ] , [ '5! ways' , 'there are 5! ways' ] , [ 'x === 5' , 'x === 5' ] ] ;
for ( const [ q , s ] of bad ) console . log ( JSON . stringify ( q ) , JSON . stringify ( s ) , checkQuote ( q , s ) . found ) ;

The first call prints found: true, start: 38, end: 76 and the original slice the UI renders. The second prints found: false with a closest passage at 0.875 (7 of 8). The third fails at 8 of 9 because "when" is not in the page; the spaces and capitals alone were forgiven. The fourth prints true, and all 18 regression lines print false.

If the closest score replaced the KMP match, the second call would pass at a threshold of 0.8; at 0.9 it still fails.

Limits: lowercasing merges "US" and "us"; a Unicode minus will not match "-0.5" (a false alarm, never a false pass); and "5" quoted from "5 ms" still matches, so show surrounding text with every quote. It covers prose with numbers and simple comparisons; treat code and formulas as "could not check" until your tokenize