Getting Clean Text Out of PDF, DOCX, and HTML

PDF: No Paragraph, No Reading Order

A PDF is a drawing program's output. The content stream says "place this glyph at these coordinates in this font." No paragraph, no reading order, no notion that this run is a footnote and that one a heading. Libraries handing you a page.get_text() string are guessing, usually "sort by y, then x." That dies on the first two-column academic paper: line one of column A, then line one of column B, then line two of column A - interleaved garbage. Fed into anything narrative, it produces confident nonsense.

Detect columns geometrically before ordering anything. Project every text block's horizontal extent onto the x-axis and look for a sustained gap:

def detect_columns ( blocks , page_width , min_gap_ratio = 0.04 ):
    """ blocks: [(x0, y0, x1, y1, text), ...]. Returns x-cut positions. """
    occupied = [ False ] * 100  # 1% buckets across the page
    for x0 , _ , x1 , _ , _ in blocks :
        lo , hi = int ( x0 / page_width * 100 ), int ( x1 / page_width * 100 )
        for b in range ( lo , min ( hi , 99 ) + 1 ):
            occupied [ b ] = True
    cuts , start = [], None
    for i , filled in enumerate ( occupied ):
        if not filled and start is None :
            start = i
        elif filled and start is not None :
            if ( i - start ) >= min_gap_ratio * 100 :
                cuts . append (( start + i ) / 2 / 100 * page_width )
            start = None
    return cuts

def reading_order ( blocks , cuts ):
    """ Within each column top-to-bottom, then columns left-to-right. """
    col = lambda b : sum ( 1 for c in cuts if b [ 0 ] > c )
    return sorted ( blocks , key = lambda b : ( col ( b ), round ( b [ 1 ], 1 ), b [ 0 ]))

Two caveats. Run this per page, not per document - papers are often single-column for the abstract and two-column afterward. And a whitespace gap can be a real column boundary or just a figure with text either side; require it to persist across most of the page height before trusting it.

Headers, Footers, and the Boilerplate You Must Delete

Every page carries running heads, page numbers, journal names, DOIs. Extracted naively, these splice mid-sentence into your text: "...the distribution is 14 Journal of Irreproducible Results, Vol. 3 skewed toward..." Once in, it's nearly impossible to remove downstream, and it will end up read aloud or embedded.

The reliable detector is repetition across pages, not position:

def find_boilerplate ( pages , band = 0.10 , min_ratio = 0.6 ):
    """ Text in the top/bottom band of >=60% of pages is chrome. """
    counts = Counter ()
    for page in pages :
        h = page . height
        for _ , y0 , _ , y1 , text in page . blocks :
            if ( y1 < h * band or y0 > h * ( 1 - band )) and len ( text ) < 120 :
                counts [ normalize ( text )] += 1  # digits -> '#', casefold
    return { t for t , c in counts . items () if c >= max ( 2 , int ( len ( pages ) * min_ratio ))}

Normalizing digits to a placeholder before counting makes page numbers collapse into one repeated pattern instead of looking unique on every page. Guard it behind a page-count minimum - on a two-page document, "appears on 60% of pages" means "appears twice," which catches real content.

Footnotes

Footnotes need similar treatment for a different reason: legitimate content, but splicing them mid-sentence into the body is wrong. Detect them by font size (typically 1-3 pt smaller than body) plus lower-region position, then pull them into a separate stream. Dropping or appending them is a product decision; extraction's job is only to not interleave them.

Tables

Tables deserve the same honesty. For ruled tables you can recover cells from the drawing operations, intersecting line segments into a grid. For borderless tables - most of them - you're inferring structure from whitespace alone, and reliability collapses with merged cells, wrapped text, and nested headers. Don't solve this generically: classify the region, attempt structured extraction, and if the result fails a sanity check (consistent column count, plausible cell density), fall back to a natural-language summary. "Table 2 compares four models across three benchmarks" beats a soup of misaligned numbers.

DOCX and HTML: Different Problems, Both Underestimated

DOCX is a zip of XML and far more tractable - paragraph structure is explicit. The traps are elsewhere: text lives in w:t runs that split mid-word at formatting boundaries ( <w:t>hyper</w:t><w:t>link</w:t> must be joined before tokenizing), tracked changes leave deleted text in w:del elements you must exclude, and text boxes and headers sit outside the main flow.

HTML from a URL is a boilerplate-stripping problem: nav, cookie banners, related-articles, comments, modals. The classic Readability-style approach scores DOM nodes on text density - text length against link density, weighted by tag semantics - then takes the highest-scoring subtree. Good default; fails predictably when the article is fragmented across many small containers.

The cheap win most people skip: check application/ld+json first. A large share of publishers emit an Article schema with articleBody populated, and when it's there it's cleaner than anything heuristics will give you.

TYPES = ( " Article " , " NewsArticle " , " BlogPosting " )
def extract_article ( html ):
    for block in find_jsonld_blocks ( html ):
        for node in flatten_graph ( json . loads ( block )):
            if node . get ( " @type " ) in TYPES and ( b : = node . get ( " articleBody " )):
                return b
    return readability_extract ( html )  # fall back to density scoring

Normalization Pass

Whatever the source, you land in the same normalization pass: de-hyphenate line-broken words (but keep genuine hyphenates - check whether the joined form is more plausible than the split), collapse the ligatures PDF fonts love (fi, fl), normalize Unicode quotes and dashes, and rebuild paragraph boundaries from line spacing rather than trusting newlines.

Building the ingestion side of DuoCast, which takes PDF/DOCX/URL input, what surprised me was how much of the perceived output quality traced back to this layer. A model handed interleaved two-column text produces plausible-sounding wrong content - and nothing in the output reveals extraction as the culprit. Which is why this stage should emit diagnostics: column count, boilerplate removed, footnote count, unresolved regions. When quality drops, that's the only way to know where to look.