Context-Aware TTS
Overview
A desktop app that turns a web-novel chapter into an audiobook where every character has their own voice. Paste a chapter URL and it scrapes the page, then works out who says each line of dialogue. Each character gets a distinct voice that stays the same across chapters.
The interesting part is not the speech synthesis, which is a solved problem. It is the attribution: given "I told you so," she said, and then three more lines with no speech tag at all, which character is speaking? That question turns out to be most of the project.
- Neve “The gate was already open when I got here,” Neve said. Has a speech tag. The only easy line here.
- Tobin “Then somebody opened it.” No tag. Only the fact that it is a reply makes it his.
- Tobin “Nobody in this house has a key to it but you, Neve.” No tag, and Tobin speaks twice running, so alternating between two speakers gets this wrong. The name in the line is who he is talking to, not who is talking.
- narrator The lock lay in the grass, sheared clean through. An action beat that names nobody and belongs to no speaker.
- Neve “It was cut. Whoever came through never needed a key.” No tag. She is answering the accusation, which is the only thing that identifies her.
- Tobin “Then we should not be standing here,” Tobin said. Tagged again, which is how a reader confirms the run above.
Everything runs locally. The grammar model, the language model and the voice model download once and then run on your own machine. It is about 18,000 lines of Rust, built with egui.
The Pipeline
Fetch. Extractors for RoyalRoad and SpaceBattles pull the chapter and flatten the HTML into typed blocks. Forum-hosted fiction has no paragraph tags at all, so a line break has to flush a paragraph instead. SpaceBattles needs a logged-in session, which the app gets by reading cookies out of the local Firefox profile.
Correct. Web fiction is full of typos, and typos make speech synthesis stumble, so each block passes through a grammar-correction model. There was no suitable ONNX build of it, so I exported one myself and published it to HuggingFace. Running it meant hand-writing the encoder-decoder decode loop, threading the key-value cache tensors between iterations by hand.
Split. Pure Rust, no model. Each story configures its own dialogue delimiter pairs, so anything inside a pair is dialogue and everything between pairs is narration. Stories vary: Super Supportive writes text messages in square brackets, so that story adds a bracket pair and its text messages start being spoken as dialogue.
Attribute, then generate. A language model running in-process on llama.cpp names the speaker of each dialogue segment, and Piper synthesises each segment in that speaker's voice. Finished audio joins the playback queue as it completes, so listening starts before the chapter has finished processing.
Attribution
The prompt gives the model the numbered segments, the story's known
characters, a window of surrounding context, and who spoke the
preceding lines. It must answer with one line per segment in the
form [N] Speaker.
That plain-text format is a workaround, not a preference. The first approach was grammar-constrained JSON, and the JSON grammar is still in the source, dead, with a comment explaining why. The grammar sampler in the llama.cpp bindings aborts the process on its first call, and the alternative constraint engine mis-handles the tokeniser's byte-pair tokens badly enough to reject valid output.
Two hard rules run after the model answers. Any segment the splitter marked as narration is forced to the narrator regardless of what the model said, and if every segment is narration the model is never called at all. Constraining what the model is allowed to be wrong about turned out to be worth more than asking it more nicely.
Characters are matched to voices by meaning rather than by rotation. An offline tool embeds each catalogue speaker's trait annotations, and at runtime the character's editable voice-traits string is embedded the same way. The app picks the nearest speaker within the requested gender, skipping any voice already taken. So "a stern, authoritative man" resolves to a specific real speaker, and no two characters in a story collide.
Architecture
Epoch invalidation. Every job carries the epoch it was submitted under, and editing a story bumps the epoch. Results arriving under a stale epoch are dropped on receipt. Nothing is ever cancelled: a six-second model call that is now pointless runs to completion anyway and its result goes in the bin. That removes the whole class of races around editing a character while work is in flight.
One state machine, no side tables. Each line moves through a single phase enum, and that enum carries the data for the phase it is in. The pipeline keeps no in-flight tracking maps at all. Concurrency budgets, elapsed times and the status shown in the UI are all derived by walking the line phases, so there is no second copy of the state to drift out of sync with the first.
Sizing itself to the machine. On first launch the app enumerates the GPU backends, takes the largest available VRAM, and picks one of four tiers. The tier chooses the quantised model, the context length, and how many characters of story context to feed the model. A bigger GPU therefore widens the context the model sees around the line it is attributing, on top of running faster.
Measuring It
Attribution quality is measured against five hand-labelled chapters, from 55 to 151 lines each. The test runs the real production path and scores against ground truth, with a pass mark of 85%. Consecutive runs of the same speaker collapse before comparison, so a splitting difference never counts as an attribution error. Because a run against a local model is slow, the harness aborts early using a Wilson bound: as soon as a chapter is statistically unlikely to reach 85%, it stops.
The engineering log is where this pays off. Five lines the model always got wrong were run twenty times each to measure success as a probability rather than a coin flip. All five started at zero. Four fixes were tried: stronger scoping instructions, explicit turn-taking rules, pronoun hints, and a block of recent attributed conversation. The first three did nothing at all across a hundred attempts each. The fourth fixed its target case twenty times out of twenty and made the overall suite worse, because the history biased the model toward continuing the pattern instead of reading the text. None of them shipped.
What worked was a bigger model. Moving from a 4B to a 12B improved every chapter, by as much as 17 points on the hardest, and took all five impossible lines to twenty out of twenty. Temperature stopped mattering entirely: two different temperatures produced byte-identical output across all five chapters. A two-pass attribution scheme was designed, written up, and then deliberately not built, because the larger model had made it unnecessary.
A later benchmark of nine models sharpened it. Model generation beat model size: a newer 4B outscored an older 4B of identical file size by five points, and a newer mid-size model matched the 12B's average at 20% smaller, which is what the app now ships.
The reason for starting again was the toolkit rather than the design. Qt and QML were unpleasant to work in, and egui was not. That preference has outlasted this project and carried into later ones. Most of the architecture came across intact; what changed was everything the toolkit had made awkward.