The Problem of Robotic Voice When Changing Speech Speed - From Phase Vocoder to WSOLA

📝 Originally published (in Japanese) at forge.workstyle.tech. While building a "Voice Design" app, I added a slider to adjust speech rate (speaking speed). The goal was simple: speed up or slow down the tempo without changing the pitch. This is a very common requirement. librosa has a function called librosa.effects.time_stretch designed exactly for this. It’s a one-liner. That’s what I used at first. However, as soon as I moved the slider even slightly, a faint metallic ringing would appear in the output. The voice sounded slightly "robotic" and "echoey," becoming muffled. The original voice was natural, but the moment the tempo changed, the quality dropped. This article is a record of how I discovered that the cause was phase blurring in the phase vocoder and how I resolved it by implementing WSOLA (Waveform Similarity Overlap-Add) from scratch using numpy . Premise: Changing tempo while preserving pitch If you simply drop or duplicate audio samples, the pitch will shift along with the playback speed (the "chipmunk effect" you get when fast-forwarding). Time stretching is the process of changing only the tempo while avoiding this pitch shift. There are two main approaches to this: - Phase Vocoder: Converts the signal into the frequency domain using STFT, then stretches/compresses it by adjusting the phase advancement of each frequency bin. It operates in the frequency domain. - WSOLA: Cuts the waveform (time domain) into short frames and finds the best positions to overlap and add them so they connect smoothly. It operates in the time domain. librosa 's time_stretch uses the former: the phase vocoder. The Symptom: Metallic ringing (Robotic voice) A phase vocoder treats each frequency bin independently and updates the phase to the "intended" advancement amount. While mathematically sound, in real speech, the phase relationships between harmonic components (overtones) gradually fall apart. This is known as a loss of phase coherence. To the human ear, this manifests as: - A metallic or electronic sound (often called "phasiness"). - A faint reverberation or echo effect. - Blurry vowel cores, making the voice sound "synthetic." While this is often unnoticeable in music or percussive material, human speech relies heavily on formants and harmonic structures. Because of this, phase blurring is heard very clearly as a "robotic voice." Furthermore, the effect worsens as the stretch ratio increases. The conclusion was that it was a poor match for a speech-rate slider. The Solution: WSOLA in the time domain WSOLA does not enter the frequency domain; it simply cuts and pastes the waveform. Instead of "recalculating" the phase, it uses cross-correlation to find the position where adjacent frames connect most naturally. Therefore, phase blurring does not occur by design. The logic works like this: - On the output side, frames are arranged at fixed intervals ( syn_hop ). - From the input side, we take frames at an interval corresponding to the stretch rate ( ana_hop = syn_hop × rate ) at the "ideal" positions. - However, instead of the exact ideal position, we search within a range of ± tol to find the frame that most closely matches the continuation of the previously placed frame (creating a natural waveform continuity). - The found frame is then combined using a Hann window (overlap-add). The heart of the implementation (_apply_speed ) is this "coarse search for the similar position." ana_hop = int(round(syn_hop * float(rate))) win = np.hanning(frame).astype(np.float32) ... # The "natural continuation" of the previous frame = nat nat = xp[off + prev_ana + syn_hop: off + prev_ana + syn_hop + frame] best_d, best = 0, -1e18 for d in range(-tol, tol + 1, 8): # Coarse search within ±tol to align phase cand = xp[off + ideal + d: off + ideal + frame + d] sc = float(np.dot(cand, nat)) / (float(np.linalg.norm(cand)) + 1e-6) if sc > best: best, best_d = sc, d a = ideal + best_d seg = xp[off + a: off + a + frame] y[syn:syn + frame] += seg * win norm[syn:syn + frame] += win Here, nat represents the "natural continuation of the previously placed frame." By shifting the candidate frame cand within the range of ±tol , we find the position best_d where the normalized dot product (cross-correlation) with nat is maximized. Essentially, we are stitching the waveform where the periods align, which guarantees phase continuity. Finally, we normalize by dividing by the window weight norm (a standard practice in overlap-add). The parameters used were frame=1024, syn_hop=512, tol=512 . I used a coarse search with a step of 8 samples (range(-tol, tol+1, 8) ) instead of a sample-by-sample exhaustive search. Since speech rate adjustment requires real-time performance, this provides a good balance between audio quality and processing speed. Trade-offs: It is not a silver bullet Switching to WSOLA does not solve everything. There are material-dependent trade-offs in method selection. - WSOLA is strong for speech (single speaker). Since periodicity is clear, it is easy to find similar positions, avoiding phase blurring. This was the best choice for this specific use case. - On the other hand, for complex polyphonic music or material with many transients, a phase vocoder may be less prone to artifacts. Since WSOLA commits to a single "stitching position," when multiple periodicities are mixed, the similarity search can get lost, leading to rhythmic fluctuations or "doubling" effects. - At extreme stretch ratios, artifacts increase regardless of the method. If you stretch too much with WSOLA, repetitions of the same frame become noticeable. It’s not a matter of "phase vocoder is bad and WSOLA is justice," but rather that WSOLA was better suited for speech, which has very clear pitch structures. Pitfalls and Lessons Learned - Question a single line of a library. librosa.time_stretch was working perfectly. It wasn't a bug; the characteristic of the phase vocoder method (phase blurring) simply didn't suit speech. The key was distinguishing between "the function is broken" and "the method doesn't fit the purpose." - Be aware of "Time Domain vs. Frequency Domain" from the start. If you can pinpoint that metallic sound as being phase-derived, you can logically conclude that you should move from frequency-domain processing (phase vocoder) to time-domain processing (WSOLA). - A custom implementation can be very small. The core of WSOLA is just "finding the similarity position via cross-correlation and performing overlap-add," which can be written with just numpy . A side benefit was reducing an external dependency. - Don't forget normalization for the window and overlap. In overlap-add, the amplitude only becomes correct when you divide by the sum of the window weights ( norm ). If you skip this, the volume will fluctuate. Summary - The "robotic voice/metallic ringing" in speech rate adjustment is actually phase blurring (phasiness) caused by the phase vocoder. - For materials with clear pitch like speech, WSOLA-which stitches waveforms at similar positions-is more natural than re-calculating phases with a phase vocoder. - The core of WSOLA is searching within $\pm$ tol for the position that has the maximum cross-correlation with the "natural continuation" of the previous frame, and then performing overlap-add with a Hann window. This can be done in a few dozen lines ofnumpy . - It is not a universal solution. For polyphonic music or transient-heavy material, a phase vocoder can be advantageous; the correct approach is to choose the method based on the source material. - Before suspecting that a library function is "broken," check if the underlying algorithm (methodology) is appropriate for your specific use case. Top comments (0)