Building an investing knowledge graph, part 6: if you're merging data from more than one source, you'll hit this

Building an investing knowledge graph, part 6: if you're merging data from more than one source, you'll hit this

Since part 5 went up, I've had two conversations worth writing about. One was with someone building a financial news aggregation system who wanted to know if the ER setup here would work for their case. The other was with someone doing KYC screening who asked roughly the same question. Those conversations are what prompted this final part.

The financial news question had an obvious answer: yes, with some caveats about cross-language blocking. The KYC question took me longer, because the error tolerance is so different that I wasn't sure how much of what I built would actually transfer.

The KYC conversation

The person I talked to was working on sanctions screening. Their problem was name matching: entity strings from incoming documents against a registry of flagged persons and organizations. The description of the ER problem was familiar enough that I recognized it immediately. What I had to think through more carefully was what the differences meant in practice.

In the investing graph, a false negative means a traversal stops short. Two nodes that should be the same entity exist as separate nodes, and a path that should go through them stops at the first one. The graph is less useful, and you can often tell something is wrong because you're getting empty results where you expect some.

In sanctions screening, a false negative is an entity that should have been flagged clearing a check it shouldn't have. The downstream consequences are not "the graph is less useful." This changes how you'd set the threshold, and it probably changes how you'd approach training data.

The model I've been running was tuned for a corpus where some missed merges are acceptable in exchange for keeping false positives low. For compliance tooling, that tradeoff goes the other way. I couldn't tell the KYC person "use the same model and threshold" and be confident they'd get results they could actually rely on.

So: the core architecture is the same. The pairwise scoring, the registry with a stable and pending boundary, the review layer for uncertain pairs. But the model would need to be retrained on labeled pairs from their actual domain, and the threshold would need to be set differently. These aren't small caveats. Blocking strategy would probably change too, because person name matching across languages is harder than company name matching in English.

The financial news question

This one was more tractable. They were ingesting from Bloomberg, a Chinese-language wire service, and some regional outlets. "Tesla, Inc." and "特斯拉" and "the EV maker" arriving in the same week.

The architecture question I could answer directly: yes, the same setup applies. The harder question was about cross-language blocking. For English company names, string normalization gets you a long way. Strip legal suffixes, lowercase, collapse whitespace, and most aliases cluster cleanly enough that phonetic blocking on the normalized form works. For Chinese-English matching, you need something else.

I spent some time thinking through what I'd actually try. One approach is alias generation during ingestion: before a string hits the resolver, you run it through a step that produces language variants and stores them as additional candidates. Then blocking happens on normalized alias strings rather than on raw input. The problem with this is that you're trusting the alias generation step, which introduces a new source of noise. I've thought about this but haven't implemented it, and I was honest with the person asking that this was speculation rather than something I'd tested.

Main takeaway from both conversations

The main takeaway from both conversations is that the registry architecture is more transferable than the model itself. The idea of maintaining a stable set of committed decisions versus a pending set of uncertain pairs under review, and serving only the stable set to external callers, holds regardless of what domain you're matching in. The specific model weights and threshold values don't hold.

What I would tell someone starting from scratch

This is the part I'm most uncertain about writing, because there's a gap between "what I learned from this specific project" and "what would generalize to your project."

The concrete thing I'd say: budget for registry maintenance in a way you probably won't. The ongoing cost of running this system is managing accumulated state, not running inference. Close to 48,000 entities and aliases have accumulated over months of ingestion. Some of those decisions were made when the threshold was still being calibrated. I know this in aggregate because I watched the model improve; I don't know which specific entries are wrong without re-running the full corpus. That re-run is still on the list.

I'd also say: the blocking step is where I lost the most time to underestimation. It's easy to focus on the model and assume that if you have good coverage you'll catch most matches. The model doesn't see pairs that didn't make it through blocking. If your blocking strategy misses a class of matches systematically, you won't know from looking at the model's output alone.

The third thing, and I've mentioned this before but it bears repeating at the end of a series: Splink is doing the real probabilistic lifting here. It's a well-maintained open-source library, and I'd recommend starting there before building anything custom.

ER API and what comes next

The investing knowledge graph is still running. The entity resolver is live on Railway as ER API, and I'm expanding access based on how early users are actually using it.

The endpoints are:

• /v1/resolve for single string resolution
• /v1/splink-pairs for batch candidate scoring

If you're running into the entity resolution problem in your own pipeline and want to try the service, there's a form at hannune.ai.

All six parts

• Part 1: From "does this news matter" to a graph I can query
• Part 2: The query SQL can't answer
• Part 3: Same company, seven names
• Part 4: Building the resolver
• Part 5: What LIVE actually means

Built on Splink for probabilistic record linkage.