Why scraping 10-K HTML gave me fabricated country revenue (and how I fixed it with XBRL)
DEV Community

Why scraping 10-K HTML gave me fabricated country revenue (and how I fixed it with XBRL)

I'm building MetricsHour, a financial data platform that tracks where public companies actually earn their revenue, by country, straight from SEC 10-K filings.

My first parser was a mistake: it scraped the rendered HTML of the filing. When I audited it, only about 9% of tracked stocks had correct and complete data. The rest had:

  • Fabricated country rows
  • Currency misreads
  • Wrong period

From HTML Guessing to XBRL Structure

Every 10-K ships with an XBRL instance document, and it already holds the structure I was trying to guess at. The rewrite pulls each field from where it's defined:

  • Currency from unitRef
  • Period from contextRef
  • Geography from explicit dimension members

I also had to add a third strategy for vertical fact tables. After that, AMT resolved 21 countries cleanly.

Where to See It

  • Browse stock pages with geographic exposure: metricshour.com/markets/
  • Open source package: edgar-geo-revenue on GitHub

It's also on PyPI and Hugging Face, and there's a Kaggle dataset.

Lessons

  • If a structured source exists, never parse the rendered version.
  • Audit against known-good tickers before trusting any pipeline.
  • Ticker drift is real. I had to fix six aliases (BLL, WRK, RE/EG, PARA, BK, MMC) and mark 10 delistings inactive.

Issues and PRs welcome. If you've worked with XBRL dimensions, I'd like to hear what edge cases you've hit.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.