Hacker News

Show HN: An API for word timestamps from speech, 75% within 20ms

We built a really good forced aligner.

For a while now, deleting a word in Yarn has had a decent chance of clipping the word next to it. (Yarn is a video editor where you, or agents, edit by editing the transcript.) Word times come from a forced aligner, and none we found was accurate enough, so we built Quebec to fix that.

It puts 77% of word edges within 20 ms of a human’s; the next best gets 54%. Here's an example before and after:

We found that a clean cut needs each word’s start and end to within about 20 ms. Getting those times is called forced alignment: you give it the audio and the transcript, and it tells you when each word starts and ends.

We tried WhisperX, ElevenLabs Cue and the Montreal Forced Aligner (MFA) in turn, then measured these amongst a few others (like Qwen3-ForcedAligner) against hand-labelled speech. Even with the best of them, around a fifth of cuts come out wrong.

We looked at two things: how many word boundaries are within 20 ms of where a person put them, and how often deleting a single word goes wrong. Our editor cuts at the midpoint of the gap to the next word, and we call it a bad cut if that takes more than 50 ms off the neighbour or leaves more than 50 ms of the deleted word behind.

WhisperX and ElevenLabs both put word starts about 50 ms late (we still don't know why). Subtracting that helps a lot but not enough, because the timings are noisy as well as late.

MFA is more precise, but it sometimes loses its place. About 1 word in 12 came back with no time at all, and now and then a whole phrase ended up somewhere it clearly wasn't said.

We figured somebody must have fixed this already, but found nothing better. Quebec is extremely close to as good as hand-labelling, and you can try it at quebec.yarn.so.

It took three things: fencing beam search with a custom decoder, a small neural model to sharpen boundaries and training data for that model.

1 The decoder

Containing MFA search with a fence. First, the fence. MFA can sometimes lose its place because of its beam search. It throws away unlikely paths as it goes, and if the audio and the transcript disagree for a second, the right path can be one it already threw away. (We first tried having an LLM referee between MFA and the transcriber. The best models picked the right one only a little better than chance.)

The transcriber (ElevenLabs Scribe for us) has sloppy timings, but it's almost never off by a lot (under 2% of words are more than 200 ms out). So we let the transcriber say roughly where each word is, and only let the precise search look near there.

That meant writing our own decoder. It uses MFA's acoustic model, but each word has to start and end within 200 ms of where the transcriber put it. Boxed in like that, the search is small enough to solve exactly, with no pruning, and long recordings can be split at pauses and run in parallel.

Words that came back with no time went from 1 in 12 to about 1 in 500, and bad cuts dropped from MFA's 21% to 12%.

2 The boundary model

A small model that only looks at boundaries. The fence mostly helped with the large misplacements, but only about 60% of boundaries were within 20 ms even with the new decoder, barely better than MFA. That was mostly a limit of the acoustic model, which is trained to say which sound each 10 ms slice of audio is. Many real world joins overlap phonemes, in which case the acoustic model can put the line almost anywhere in the blend and score about the same.

To solve this, we added a small convolutional net that looks at each boundary on its own. It gets 200 ms of audio on each side, as a spectrogram plus features from the first two layers of Whisper tiny, and says where in that window the boundary is, in 5 ms steps. It can move a boundary by at most 50 ms.

This works because by this point the problem is small. The fence already has nine boundaries in ten within 50 ms, so the net has a contained problem of deciding where one sound hands over to the next. It takes boundaries within 20 ms from 60% to 77% and the median error from 15 ms to 8 ms.

We trained it on about 73,000 labelled boundaries from public and private recordings. We were able to build the private dataset surprisingly efficiently with LLMs.

3 Training data

Getting 'hand-labelled' data. The net needs speech with hand-marked boundaries to learn from, and nearly all of that is paid or research-only. The one openly licensed set we found was the Map Task corpus, about 56,000 boundaries. We need more data that was diverse and matched our varied customer recordings so we tried labelling with LLMs.

The first few tries didn't work, showing a model a spectrogram and asking for the boundary was worse than the decoder's own guess. Coding agents drawing their own plots burned about 100,000 tokens per 15 boundaries and were no better.

The version that worked was one Gemini call with the decoder's guess for where to look, two plots of the audio at different zooms, and a page of rules about where words start and end. One of the rules:

For a word ending in a vowel, nasal or liquid, the word ends where the regular strong glottal pulses stop and the 500-4000 Hz energy collapses [...] the weak decaying low-frequency murmur [...] belongs to the pause.

We checked the labels a few ways before training on them. Against human labels Gemini was within 20 ms 69% of the time, where the guesses it started from were at 51%. Two different Gemini models were within 20 ms of each other 94% of the time. And adding Gemini's labels to the training set helped as much as adding human labels for the same boundaries, 73.9% against 74.0%.

We labelled 16,852 boundaries from recordings made with our product. It took a couple of hours and cost under $200.

Results

All this put together gives us an extremely accurate forced aligner for english speech, and results that transfer well across datasets and labelling methods. Here are results for a corpus of American speech that was never seen in training:

Buckeye corpus Boundaries within 20 ms Bad cuts
Quebec 75% 11%
MFA 61% 19%
ElevenLabs 30% 39%
WhisperX 24% 49%

Here's a visual of a delete like the one at the top, with ElevenLabs' timings and with ours.

Speed

It also took some work to get our aligner to be fast and scalable. We run our aligner on Lambdas, and long recordings are cut at pauses into 2-minute pieces that run in parallel, so long recordings aren't much slower than short ones.

Our decoder is about twice as fast as the Kaldi search it replaced, because the fence leaves it so little to search. And we switched from HuBERT to using the first two layers of Whisper tiny for our boundary model to get latency down on lambda CPU. Our boundary model now runs in 0.7s per minute of audio.

Caveats

We didn’t train an aligner from scratch. The acoustic model is from MFA, we start from the transcriber’s rough timings and the features we train on are Whisper’s. Our work is in how they’re combined and training on our generated boundary labels.

It’s English only, because that’s what our customers speak. Nothing in the approach is specific to English, but we haven’t tried it elsewhere.

We tradeoff speed for quality on short recordings. ElevenLabs aligns 4 minutes of audio in about a second and we take about 7s. This doesn't scale since we parallelize chunks.

Try it

Quebec is live, and the docs are below. You can send it audio and get back a transcript with word times, or send audio with a transcript you already have and get times for those words. There’s no signup, and it’s free until 8 pm Eastern on Monday. Point your agent at this page to get started. If you know of a better aligner, we'd love to try it.

Docs

The same docs are plain text at quebec.yarn.so/llms.txt.

It costs $0.32 per hour of audio, billed by the second. Failed requests are free. English only, up to 60 minutes and 1 GB per file. An hour of audio takes about a minute.

  1. Create a key
curl -X POST https://quebec.yarn.so/keys

-> 201 {"api_key": "qb_123_...", "balance_usd": 0, "top_up_url": "https://quebec.yarn.so/top-up/123"}

There is no signup. Save the key somewhere secret, because it is shown only once. If you lose it, email h***@yarn.so from the address you verified or paid with and we'll move the balance to a new key.

  1. Add credit

Open top_up_url in a browser to pay $10 by card. Add ?amount=25 or ?amount=100 to pay more. /balance gives you the link again.

A company email gets $5 free:

curl https://quebec.yarn.so/credit -H "Authorization: Bearer qb_123_..." \
-H "Content-Type: application/json" -d '{"email": "j*@acme.com"}'

-> 202 {"status": "verification_sent"}

We email a link, and the $5 is added when it's opened. Once per company, and not for Gmail or other free addresses.

  1. Transcribe
curl https://quebec.yarn.so/transcribe -H "Authorization: Bearer qb_123_..." -F file=@talk.mp3

-> 200 {"text": "Hello, there.", "duration": 2.0, "words": [{"text": "Hello,", "start": 0.43, "end": 0.88, "speaker": "speaker_0"}, ...], "cost_usd": 0.000178, "balance_usd": 9.999822, "timings": {"transcribe": 1.2, "align": 0.9, "total": 2.1}}

Any common audio or video format works. Times are in seconds.

  1. Or align a transcript you already have
curl https://quebec.yarn.so/align -H "Authorization: Bearer qb_123_..." -F file=@talk.mp3 \
-F text="Hello, there."

-> 200 {"duration": 2.0, "words": [{"text": "Hello,", "start": 0.43, "end": 0.88}, ...], "cost_usd": 0.000178, "balance_usd": 9.999822, "timings": {"total": 1.9}}

text is the plain transcript, with no speaker labels or timestamps. Each word comes back with its times, in order.

  1. Check the balance
curl https://quebec.yarn.so/balance -H "Authorization: Bearer qb_123_..."

-> 200 {"balance_usd": 9.999822, "top_up_url": "https://quebec.yarn.so/top-up/123"}

Errors look like {"error": {"code": "...", "message": "..."}}.

  • 401 invalid_key The Authorization header is missing or the key is unknown.
  • 402 insufficient_balance The balance is $0 or less. error.top_up_url is the link to add credit.
  • 400 missing_file No form field named file.
  • 400 missing_text /align needs the transcript in a form field named text.
  • 400 too_many_fields More than 5 form fields besides the file.
  • 400 unreadable_audio The file could not be read as audio.
  • 400 invalid_amount ?amount= must be 10, 25 or 100.
  • 400 invalid_email /credit needs JSON like {"email": "j*@acme.com"}.
  • 404 not_found No such route, or no account for that top-up link.
  • 409 not_eligible error.reason says why: free_email_domain, key_already_credited, domain_already_credited, recently_sent (the company got a link in the last day) or key_recently_sent (this key did).
  • 409 credit_unavailable Free credit is switched off for now.
  • 413 too_long Over 60 minutes of audio.
  • 413 too_large Over 1 GB of audio, or over 1 MB of text.
  • 413 too_much_text /align got more than 10 words per second of audio.
  • 429 busy We're at capacity. Wait Retry-After seconds and try again.
  • 502 failed The request failed and nothing was charged. Try again.
  • 500 internal_error Something else went wrong. Try again.

Notes: Map Task is 212 minutes of the HCRC Map Task corpus (Anderson et al. 1991, CC BY 4.0), and every system aligned the same ElevenLabs Scribe transcript. MFA is 3.x on default settings, Gemini is Gemini 4 Argon, and Buckeye is used only for testing. Quebec's training data includes other Map Task speakers, none of them in the test set. The 79% is from Goldman 2011, on a different corpus.

Read on Hacker News ↗ ← Back to News

Comments

No comments yet. Start the discussion.