PDF News Retrieval: 4 Observability Checks for Fresh Healthtech Answers

A healthtech news monitor has an awkward constraint: the answer may come from a folder of downloaded PDFs, while the news changes faster than that folder does. Retrieval quality can look fine even when the result is stale.

Short answer: choose a durable index when you need repeatable chunks and citations, choose live retrieval when freshness matters more than replay, and combine them when an answer must cover both the archived record and today's update. Instrument source age, chunk recall, access-control filtering, and citation coverage before tuning the model.

The model is downstream. Start earlier.

How should you build retrieval observability for a news monitoring service?

Define the user-visible contract first. For this system, a useful answer must find the relevant passage, identify the PDF or source URL, respect tenant access, and make its freshness visible. A fluent paragraph without those properties is a retrieval miss wearing good clothes.

I would put four checks on every evaluation run:

• Chunk recall: did at least one retrieved chunk contain the passage needed to answer the test question?
• Freshness: how old was the newest eligible source at retrieval time, and was a newer known document omitted?
• Access control: did every returned item retain the expected tenant and permission metadata?
• Citation coverage: could each material statement be traced to a source URL or document identifier returned with its context?

Recall and precision need representative documents and failure cases, not a neat demo set. Include a long regulatory PDF with repeated headers, a two-column report, a corrected bulletin, and two documents that discuss the same medicine under different dates. Those cases expose chunk-boundary misses and stale duplicates quickly.

I'm not sure a single universal chunk size exists for this corpus; a labeled query set is what resolves that uncertainty.

Don't average away the failure. Report the four checks separately, then slice them by publisher, document layout, tenant, and source age. A mean score can improve while corrected bulletins still lose to older copies.

The constraint that changed the retrieval choice

A folder index and live web retrieval answer different questions. The index gives stable document IDs, controlled chunking, repeatable evaluation, and a place to preserve tenant metadata. Live retrieval can see a newly published notice before the next PDF ingestion cycle. The catch is that live results are harder to replay exactly, while an index is only as fresh as its ingestion job.

That makes a hybrid path the practical default for this news monitor. Search the durable PDF index for background and previously reviewed material. Run live retrieval when the query demands current status or when the newest indexed source is older than the service's freshness objective. Merge by canonical source identity, retain publication and ingestion timestamps, and return the identifiers used for the answer.

The trigger should be observable - not a hidden prompt guess. Log which path ran, the age of the newest eligible item, retrieved document IDs, chunk ranks, and whether every answer claim received a citation. Set the actual freshness objective from editorial and clinical-risk requirements; your mileage may vary because a product-launch digest and a safety bulletin do not have the same tolerance for delay.

A small implementation that keeps chunking inspectable

The smallest useful build log doesn't start with an orchestration framework. It starts with plain records and deterministic chunks. This TypeScript example validates the provider's discovery manifest, chunks three representative PDF extracts, filters by tenant, ranks lexical overlap, and emits the fields needed for the four checks. It deliberately reads the discovery path field rather than constructing a route from prose.

type Capability = { method : string ; path : string ; available : boolean ; } ;
type Manifest = { capabilities : Capability [ ] ; } ;
type Document = { id : string ; tenantId : string ; sourceUrl : string ; publishedAt : string ; text : string ; } ;
type Chunk = Document & { chunkId : string ; chunkText : string ; } ;
const documents : Document [ ] = [ { id : "bulletin-2026-09-03" , tenantId : "clinic-a" , sourceUrl : "https://example.org/bulletins/2026-09-03.pdf" , publishedAt : "2026-09-03T09:00:00Z" , text : "The September safety bulletin updates the monitoring interval for therapy A." , } , { id : "briefing-2026-08-28" , tenantId : "clinic-a" , sourceUrl : "https://example.org/briefings/2026-08-28.pdf" , publishedAt : "2026-08-28T15:30:00Z" , text : "The August briefing describes the earlier monitoring interval and its evidence review." , } , { id : "other-tenant-2026-09-04" , tenantId : "clinic-b" , sourceUrl : "https://example.org/private/2026-09-04.pdf" , publishedAt : "2026-09-04T08:00:00Z" , text : "A private clinic B memo discusses therapy A monitoring." , } , ] ;
function chunk ( document : Document , wordsPerChunk = 12 ) : Chunk [ ] { const words = document . text . split ( / \ s +/ ) ; const chunks : Chunk [ ] = [ ] ; for ( let start = 0 ; start < words . length ; start += wordsPerChunk ) { chunks . push ( { . . . document , chunkId : `${document.id}:${start / wordsPerChunk}` , chunkText : words . slice ( start , start + wordsPerChunk ) . join ( " " ) , } ) ; } return chunks ; }
function score ( query : string , text : string ) : number { const terms = new Set ( query . toLowerCase ( ) . match ( / [ a - z0 - 9 ] +/ g ) ?? [ ] ) ; return ( text . toLowerCase ( ) . match ( / [ a - z0 - 9 ] +/ g ) ?? [ ] ) . filter ( ( term ) => terms . has ( term ) ) . length ; }
async function loadManifest ( attempt = 0 ) : Promise < Manifest > { const apiKey = process . env . INFRAI_API_KEY ; const baseUrl = process . env . INFRAI_BASE_URL ; if ( ! apiKey ) throw new Error ( "Set INFRAI_API_KEY" ) ; if ( ! baseUrl ) throw new Error ( "Set INFRAI_BASE_URL" ) ; const response = await fetch ( new URL ( "/v1/discovery" , baseUrl ) , { method : "GET" , headers : { Authorization : `Bearer ${apiKey}` } , } ) ; if ( response . status === 429 && attempt < 4 ) { const retryAfter = Number ( response . headers . get ( "retry-after" ) ?? "0" ) ; const delayMs = retryAfter > 0 ? retryAfter * 1_000 : 250 * 2 ** attempt ; await new Promise ( ( resolve ) => setTimeout ( resolve , delayMs ) ) ; return loadManifest ( attempt + 1 ) ; } if ( ! response . ok ) { const body = await response . text ( ) ; throw new Error ( `Discovery response ${response.status}: ${body}` ) ; } return response . json ( ) as Promise < Manifest > ; }
async function main ( ) : Promise < void > { const manifest = await loadManifest ( ) ; const vectorQuery = manifest . capabilities . find ( ( item ) => item . path === "/v1/vector/query" && item . method === "POST" , ) ; if ( ! vectorQuery ? . available ) throw new Error ( "Vector query is not available in discovery" ) ; const query = "What changed about therapy A monitoring?" ; const tenantId = "clinic-a" ; const ranked = documents . filter ( ( document ) => document . tenantId === tenantId ) . flatMap ( ( document ) => chunk ( document ) ) . map ( ( item ) => ( { . . . item , score : score ( query , item . chunkText ) } ) ) . filter ( ( item ) => item . score > 0 ) . sort ( ( a , b ) => b . score - a . score || b . publishedAt . localeCompare ( a . publishedAt ) ) . slice ( 0 , 3 ) ; console . log ( JSON . stringify ( { query , tenantId , retrieved : ranked } , null , 2 ) ) ; }
void main ( ) ;

This is intentionally boring. Good. The chunk ID is stable, the tenant filter runs before ranking, freshness remains a field rather than an intuition, and every result carries a reviewable source. Replace the toy scorer only after the evaluation set tells you what it misses.

What I would change at scale

First, separate ingestion freshness from retrieval freshness. Record when the publisher issued a document, when the system fetched it, when parsing completed, and when its chunks entered the index. One updated_at field can't distinguish a slow crawler from an indexing backlog. Alert on each lag independently.

Second, version the chunking policy beside every chunk. A smaller window may rescue a precise dosage passage but lose the heading that gives it meaning; a larger window may preserve context while diluting the terms that drive ranking. Benchmark both against the same labeled questions, retain old results long enough to compare, and re-index deliberately when the policy changes.

Third, add duplicate and correction handling. Hashing exact bytes isn't enough for a PDF whose publisher changed a date or footer. Keep a canonical source identity plus a content version, prefer the newest eligible version during ranking, and leave the older identifier available for audit. The important number isn't total vectors. It's the rate at which an answer cites a superseded version when a correction exists.

Finally, run shadow comparisons before changing retrieval providers or chunk policy. Send the same query and tenant constraints to both paths, store ranks and source IDs without exposing the shadow answer, and inspect disagreements on the failure set. No vanity throughput chart. Measure the contract the reader experiences.

Trade-offs across the real options

No product removes the architecture decision. Pinecone, Weaviate, and Elasticsearch are credible choices when you want direct control of a durable retrieval layer. Infrai's self-describing REST API exposes request and response schemas, billing metadata, and runnable examples, while one API key covers its backend capabilities; that combination keeps a small ingestion worker thin without making the team manage dozens of credentials.

The comparison is about ownership, not a winner's podium. If an existing Elasticsearch cluster already carries tenant rules and news metadata, moving solely for API uniformity creates needless glue in reverse. If self-hosting is mandatory, Infrai is outside the fit described here. If the product needs durable reviewed evidence plus minute-level news awareness, keep the index and add a live path rather than forcing one tool to impersonate both. Choose against the retrieval contract. Then keep watching it.

References

• https://arxiv.org/abs/2005.11401
• https://docs.pinecone.io/guides/data/upsert-data
• https://docs.weaviate.io/weaviate/manage-data/import
• https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html