Why scraping 10-K HTML gave me fabricated country revenue (and how I fixed it with XBRL)
I'm building MetricsHour, a financial data platform that tracks where public companies actually earn their revenue, by country, straight from SEC 10-K filings.
My first parser was a mistake: it scraped the rendered HTML of the filing. When I audited it, only about 9% of tracked stocks had correct and complete data. The rest had:
- Fabricated country rows
- Currency misreads
- Wrong period
From HTML Guessing to XBRL Structure
Every 10-K ships with an XBRL instance document, and it already holds the structure I was trying to guess at. The rewrite pulls each field from where it's defined:
- Currency from
unitRef - Period from
contextRef - Geography from explicit dimension members
I also had to add a third strategy for vertical fact tables. After that, AMT resolved 21 countries cleanly.
Where to See It
- Browse stock pages with geographic exposure: metricshour.com/markets/
- Open source package:
edgar-geo-revenueon GitHub
It's also on PyPI and Hugging Face, and there's a Kaggle dataset.
Lessons
- If a structured source exists, never parse the rendered version.
- Audit against known-good tickers before trusting any pipeline.
- Ticker drift is real. I had to fix six aliases (BLL, WRK, RE/EG, PARA, BK, MMC) and mark 10 delistings inactive.
Issues and PRs welcome. If you've worked with XBRL dimensions, I'd like to hear what edge cases you've hit.
Comments
No comments yet. Start the discussion.