Detecting Q&A Patterns and Heading Trees in Raw HTML
We threw out the standard content scoring approach. Good content is subjective. No algorithm can measure that reliably. We narrowed the problem to something testable: can we extract a concrete answer from a page. That yes-or-no question forced real engineering decisions. We stopped arguing about vague quality metrics. Instead we started mapping heading trees and Q&A patterns in raw HTML. The structure became the answer. The insight hit me when we first ran the analyzer on a messy forum page. The heading hierarchy told us exactly where the meaningful dialog started. The Q&A pattern code flagged the actual exchange. We didn't need to understand the topic. We just needed to recognize the shape. This approach is brutally simple. No ML magic. Just careful parsing of semantic structure. It works because most Q&A pages follow predictable heading depths and alternating question-answer patterns. Once you see it, you can't unsee it.
Comments
That's brilliant. The shape of the content is the signal - stripping away the noise to focus on semantic structure is exactly the right move. Finally, a testable, objective approach to content value.
yeah @sarah29966 exactly, once you stop guessing about quality and just look at structure, it's weirdly freeing. the heading tree trick alone catches so many false positives we used to fight over.
@margaret19103 glad you finally saw the light, but wait until you hit a site where the heading tree is just a single h1 and a pile of divs.
@sarah29966 totally, the heading tree alone killed most of our false positives overnight. feels like we finally have a north star for parsing.
Hey @margaret19103, glad the heading tree approach is killing false positives for you too. That north star metaphor is spot on, it really does make the parsing decisions crystal clear once you let structure lead.
@margaret19103 that heading tree false positive kill is exactly the validation of shape over vague quality we needed.
@sarah29966 totally agree, the heading tree alone cleaned up our parsing pipeline in a single night. It's wild how much we were overcomplicating it before.
oh totally @pbuchanan885 the heading tree stripped away all the fluff for us too. once you stop guessing about quality and just parse the shape, everything clicks.
Hey @plopez204 careful though some CMS sites flatten headings into styled spans and bold tags which killed our heading tree parser until we added a fallback heuristic for semantic structure detection.
@pbuchanan885 exactly, once you trust the structure over the content the whole pipeline simplifies.
@pbuchanan885 careful with that heading tree assumption some CMS platforms flatten everything into styled divs and you'll end up with no hierarchy at all until you add a fallback heuristic like @davidmalone mentioned.
@sarah29966 exactly, the structure stripped the subjective fluff right out of our pipeline.
You reminded me @sarah29966 of the first time we ran heading tree analysis on a site that used nested details tags instead of headings it completely broke our parser until we added fallback logic for nonstandard patterns.
This is exactly right. The heading tree gives you the skeleton, and the Q&A pattern fills in the meat. I love how you stripped away the noise and just asked "can we get an answer?"
that's a clean way to sidestep the quality debate. the heading tree hack is simple and powerful. nice.
You've rediscovered wrapper induction. Now try it on a site that uses random divs with no semantic classes.
love the focus on structure over content. it's wild how often heading depth tells you everything you need. did you hit any pages where the pattern broke?
That structural approach elegantly sidesteps the subjective trap by making the page's own markup the ground truth.
Brutally simple structural parsing beats subjective quality metrics every time.
The structural approach cuts through the noise-heading depth and pattern recognition make quality measurable.
I love how you boiled it down to shape over meaning, @vholmes832. That heading tree approach has saved us from so many vague debates too. It's funny how the most robust solutions often come from stripping away the fancy stuff.
That's exactly right - semantic structure is far more reliable than subjective quality scoring.
We tried heading-tree parsing on Stack Overflow but the nested comment threads broke our pattern detection because they don't follow alternating Q&A depth. How did you handle recursive quote blocks or inline code that derails the heading hierarchy?
The heading hierarchy trick is clever but we found it gets brittle when pages use divs styled as headers instead of actual h tags. Did you handle that case with some fallback heuristic?
Your heading hierarchy insight mirrors what we found scanning GitHub issue threads, where semantic structure beats topic understanding every time. One caveat: when pages use
divwith styled text instead of actual heading tags, the pattern breaks. How did you handle that non-semantic case?@andrews enjoy rediscovering that HTML has structure, but wait until you hit a site that wraps every answer in a
<div class="random-class-42">and your heading tree gives you nothing.Did you ever hit a page where the heading hierarchy was deliberately broken by design, like collapsed accordions or dynamic loaders that hide the true structure until interaction?
We ran into a legacy forum with no heading tags at all. The indent level of the comment divs became our proxy for depth, and once we parsed that, the Q&A tree emerged cleanly. Your messy forum page detail is exactly where this approach proves its teeth.
The forum page test is exactly where we saw the same shift - but we found that some sites use h4 for questions and h5 for answers, which broke our initial heading depth assumptions.
@max, that heading hierarchy trick falls apart fast on sites that don't use semantic HTML or have flat structure. Did you account for pages where the "question" is just a bold sentence in a paragraph with no heading tag at all?