Why Speech-to-Text Is More Than Calling an AI API

A speech-to-text demo can be wonderfully small: const result = await provider.transcribe(audioUrl); return result.text; That code is enough to prove that a model can recognize speech. It is nowhere near enough to prove that a user can trust the result. Give the demo a two-hour interview instead of a 20-second clip. Let the upload lose its connection at 87%. Let the provider send the same webhook twice. Let two words overlap by 80 milliseconds, then let the next word arrive ten seconds out of order. Let the user correct a name while an AI summary is waiting in a queue. At that point, speech-to-text stops being an API integration. It becomes a document system, a media pipeline, and a distributed-systems problem wearing headphones. While building Echoryte, a workspace that turns recordings into editable, time-linked transcripts, we found a useful way to frame the problem: A speech API returns a hypothesis. A transcription product must turn that hypothesis into a durable, navigable, and explainable document. This article is not a comparison of speech models. Models and pricing change too quickly for that to age well. Instead, it is a map of the engineering boundaries that remain regardless of which provider you call. The real pipeline The naive architecture has two boxes: recording -> speech API -> text A useful system looks closer to this: upload -> inspect the actual media -> prepare a stable audio derivative -> reserve work and enqueue a job -> select a compatible provider/model -> submit an attempt -> wait through webhook and polling paths -> validate and normalize the result -> publish a transcript revision -> edit, search, translate, summarize, and export Every arrow is a place where state can be lost, repeated, corrupted, or made ambiguous. The important shift is to treat each boundary as a contract rather than a convenient function call. 1. A file name is a claim, not evidence A user uploads interview.mp3. The browser reports audio/mpeg. It is tempting to trust both. Neither is authoritative. Extensions can be wrong. MIME types are supplied by clients. Containers may hold several audio tracks, no audio track, damaged timestamps, or codecs your downstream provider cannot decode. Long recordings may also be variable-bitrate media whose reported duration is not what you expect. Before transcription, inspect the bytes with a media probe such as ffprobe and answer concrete questions: - What container is this really? - Is there a usable audio stream? - What is the measured duration? - Which track should be transcribed? - Can the media be decoded within your CPU, memory, disk, and time budgets? In Echoryte's pipeline, ingestion produces a provider-friendly audio derivative and a browser-friendly playback derivative. Doing this once creates a stable input for both recognition and later review. This also improves error messages. “The API failed” is not useful. “This file contains no audio stream” or “the measured duration exceeds your plan” tells the user what to do next. A good rule is: Validate business limits against inspected media, not against metadata declared by the client. 2. A transcription job is not a provider request This distinction sounds academic until the first retry. We model three separate things: type TranscriptionJob = { id: string; fileId: string; requestedTier: "fast" | "standard" | "precision"; languageHint?: string; diarization: boolean; }; type TranscriptionAttempt = { id: string; jobId: string; provider: string; model: string; providerJobId?: string; status: "submitted" | "waiting" | "succeeded" | "failed" | "ignored"; }; type TranscriptPublication = { jobId: string; version: number; revision: number; objectKey: string; }; The job represents user intent: “transcribe this recording with these capabilities.” An attempt represents one execution of that intent against one provider and model. A publication represents the result that users are allowed to see and edit. Separating them gives you several useful properties: - A transient provider failure can create another attempt without inventing another user job. - A late callback can be attached to the correct historical attempt. - Cancellation can ignore an old result instead of publishing it accidentally. - Retranscription can keep the existing transcript readable until the replacement succeeds. - Cost and latency can be measured per attempt rather than guessed per job. The last point matters more than it appears. If a provider succeeds but your worker crashes before publication, the provider request happened, the cost happened, and the user-visible result did not. One status field cannot describe all three facts. 3. “Use provider B if provider A fails” is not a routing strategy Speech providers do not have one-dimensional capability. Support varies by model, language, automatic language detection, word timestamps, speaker diarization, latency, and sometimes region. A fallback that returns plain text when the product promises word-level navigation is not a fallback. It is a silent contract change. A better router first removes incompatible candidates and only then ranks the survivors: function isCompatible(model: ModelCapability, request: RequestFeatures) { const language = resolveLanguage(model, request.languageHint); return ( model.tiers.includes(request.tier) && language.supportsWordTimestamps && (!request.diarize || language.supportsDiarization) && (request.languageHint || language.supportsAutomaticDetection) ); } const candidates = capabilitySnapshot.models .filter(model => isCompatible(model, request)) .sort(rankByQualityLatencyAndCost); Notice the capability snapshot. Provider documentation, model IDs, and language support change. Versioning the routing data lets you answer a difficult operational question later: “Why did this job choose that model on that day?” Failover should use the same compatibility filter and exclude capabilities already attempted. Otherwise, a retry loop can bounce between equivalent failures or quietly drop a requested feature. 4. The happy path is asynchronous; the failure paths are more asynchronous Long recordings do not belong in an HTTP request-response cycle. They need durable background work. That introduces at-least-once behavior almost everywhere: - A queue can deliver the same job again. - A worker can stop after an external side effect but before saving local state. - A webhook can be delayed, duplicated, or arrive out of order. - A polling request can race with a webhook. - The user can cancel while the provider is still processing. Our useful mental model is a state machine backed by the database: queued -> claimed -> submitted -> waiting -> normalizing -> published | | | | +------------+----------+------------+-> failed / ignored The queue schedules work; it is not the source of truth. When a worker claims a job, it receives a lease token and expiry. It renews the lease while doing slow work. Every later write checks the same token. If the lease is lost, that execution can no longer publish. This prevents an old worker from waking up after a pause and overwriting the result produced by a newer worker. External operations also need stable identities. Creating an attempt, attaching a provider job ID, storing a raw result, reserving a transcript version, and publishing should all be safe to replay. For callback-based providers, the webhook handler should do very little: - Authenticate the callback. - Record or mark the event idempotently. - Wake the durable job. - Return quickly. Polling remains useful as a recovery path when a callback never arrives. Webhooks reduce latency; polling closes the reliability gap. One more subtle point: progress is a user-interface estimate, not provider truth. A bar that moves smoothly to 73% does not mean 73% of the words exist. Show stages and historical time ranges, and label estimates as estimates. 5. Provider JSON is untrusted input Even a successful provider response is external data. Parse it at runtime. A TypeScript interface cannot reject NaN, negative timestamps, missing fields, invalid confidence values, or a word whose start is after its end. A runtime schema can. After validation, normalize the result into a provider-independent structure: type Word = { text: string; startMs: number; endMs: number; confidence?: number; }; type Segment = { id: string; speakerId: string | null; startMs: number; endMs: number; words: Word[]; }; type Transcript = { version: number; durationMs: number; language: string; speakers: Speaker[]; segments: Segment[]; }; Normalization is not just renaming fields. It is where you define what “valid time” means. For example, Echoryte's current normalizer tolerates a small, bounded word overlap by clipping the next start time. It rejects a large overlap or a word that reverses the timeline. It normalizes Unicode, removes control characters, and refuses words that become empty. Segments are then built around editing and reading behavior, not around arbitrary provider paragraphs. A speaker change forces a boundary. So does a meaningful silence. Very long segments are split at safe word boundaries. The principle is more important than the exact thresholds: Repair only what you can repair without changing meaning. Reject ambiguity before it becomes durable data. Silently sorting wildly disordered words may produce JSON that passes a schema, but it can make clicking a quote jump to the wrong moment. That is worse than an explicit failure. Keeping the original provider result separately is also valuable. You can re-run normalization after improving your rules, investigate disputes, or compare provider behavior without calling the API again. 6. The product is a time-linked document, not a string Plain text throws away the most valuable part of a transcript: its relationship to the recording. Once every word has time, several product behaviors become possible: - Click a word to seek the audio. - Highlight words during playback. - Find a quote an