Multi-Voice Audiobooks: The Character Attribution Problem

The attribution problem

Turning a novel into a multi-voice audiobook sounds like a TTS problem. It is mostly a parsing problem, and the parsing is harder than the synthesis. Attribution is not solved by quotation marks. The naive approach: find text in quotes, that is dialogue, assign a voice. It breaks immediately on real prose.

"I told you," she said, "and you did not listen."

One utterance, split across two quoted spans, with the attribution wedged in the middle. Naive splitting produces two separate lines and often assigns them different speakers.

Two hard cases

Then there is the harder case - no attribution at all:

"Are you coming?"
"In a minute."
"You said that an hour ago."

Three lines, zero speaker tags. A human tracks the alternation from context. A parser has to model turn-taking, and it has to know when the alternation breaks because a third character walked in.

What actually helps

Attribution verb lexicons beat regex. said / asked / whispered / muttered plus their inflections cover the majority of tagged dialogue. The pattern is usually "..." <verb> <entity> or <entity> <verb>, "..." - worth handling both orderings explicitly rather than hoping one regex catches everything.

Coreference is unavoidable. "she said" only resolves if you know who "she" is in this scene. Without a coref pass you get correct attribution to the wrong voice, which sounds worse than a monotone read because listeners notice the inconsistency.

Alternation as a prior, not a rule

In untagged runs, alternating between the last two active speakers is right most of the time. Treat it as a default that named attribution overrides - not as ground truth.

The consistency constraint nobody warns you about

Once a voice is assigned to a character, it must stay assigned. Chapter 14 cannot re-cast the protagonist because your attribution confidence dipped. That means voice assignment is a document-level decision made after full-text analysis, not a streaming one made per chapter. Which in turn means you cannot start rendering audio until you have parsed the entire book - an architectural constraint that surprises people building this incrementally.

Practical shape that works

• Full-text parse -> character inventory with mention counts
• Assign voices to the top N by frequency, narrator to everything else
• Persist the mapping
• Render chapters against the frozen mapping

I built this pipeline into VoxForge - epub or txt in, auto-cast characters, chaptered audiobook out.

Where it still fails

• First-person narration where the narrator is also a character.
• Epistolary novels.
• Books that switch POV mid-chapter without a section break.

All three break the character-inventory assumption, and I have not found a clean general solution - currently they need manual overrides.