Build a Voice Assistant with Python and Whisper

Why Whisper and Python?

Whisper is OpenAI’s general-purpose speech recognition model. It’s trained on diverse audio data and supports multiple languages, making it far more robust than older tools like SpeechRecognition’s built-in Google API. Unlike many cloud-based alternatives, Whisper can run locally, which means:

• No API fees for transcription
• Zero latency for network roundtrips
• Full privacy: your audio never leaves your machine
• Customizability: you can tweak the model, add wake words, or integrate with any LLM

Python is the ideal language for this because of its rich ecosystem: openai-whisper for transcription, pyttsx3 for speech output, and SpeechRecognition for microphone input.

Setting Up Your Environment

Before writing code, you need the right tools. Create a project directory and set up a virtual environment to keep dependencies isolated:

mkdir voice-assistant && cd voice-assistant
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Now install the core packages:

pip install openai-whisper SpeechRecognition pyttsx3 python-dotenv

If you run into installation issues with openai-whisper (common on some systems), swap it for the faster, more compatible faster-whisper:

pip install faster-whisper

Create a .env file to store your API keys securely (though Whisper itself doesn’t require one for local use):

# .env
OPENAI_API_KEY=your-key-here  # Optional, only if you use GPT later

Building the Core: Speech-to-Text with Whisper

The heart of your assistant is the ability to convert spoken words into text. Whisper handles this elegantly. Here’s a minimal function that loads the model and transcribes an audio file:

import whisper

# Load the Whisper model (choose "base" for speed, "large" for accuracy)
model = whisper.load_model("base")

def transcribe_audio(audio_path: str) -> str:
    result = model.transcribe(audio_path)
    return result["text"]

This function takes an audio file path (e.g., recorded.wav) and returns the transcribed text. The "base" model is fast (0.5s for 5s audio on an M2 MacBook) and lightweight (74MB), while "large" offers higher accuracy but requires more resources.

Capturing Voice Input

To capture audio from your microphone, use the SpeechRecognition library. It provides a simple interface to record and save audio:

import speech_recognition as sr
import wave

def record_audio(filename: str = "input.wav", duration: int = 5):
    recognizer = sr.Recognizer()
    microphone = sr.Microphone()
    with microphone as source:
        print("Listening...")
        recognizer.adjust_for_ambient_noise(source, duration=1)
        audio = recognizer.listen(source, timeout=duration)
    # Save audio as WAV
    with wave.open(filename, 'wb') as f:
        f.write(audio.get_wav_data())
    print(f"Saved audio to {filename}")
    return filename

This function records 5 seconds of audio, saves it as input.wav, and returns the file path for transcription.

Turning Text Back into Speech

Once Whisper transcribes your voice, you need to respond. pyttsx3 is a cross-platform text-to-speech library that works offline:

import pyttsx3

def speak(text: str):
    engine = pyttsx3.init()
    engine.say(text)
    engine.runAndWait()

This function speaks the given text using your system’s default voice. You can customize speed, voice, and language by adjusting engine properties.

Putting It All Together: The Full Assistant

Now, combine everything into a working voice assistant. Here’s a complete script that records, transcribes, and responds:

import whisper
import speech_recognition as sr
import wave
import pyttsx3

# Initialize Whisper
model = whisper.load_model("base")

def transcribe_audio(audio_path: str) -> str:
    result = model.transcribe(audio_path)
    return result["text"]

def record_audio(filename: str = "input.wav", duration: int = 5):
    recognizer = sr.Recognizer()
    microphone = sr.Microphone()
    with microphone as source:
        print("Listening...")
        recognizer.adjust_for_ambient_noise(source, duration=1)
        audio = recognizer.listen(source, timeout=duration)
    with wave.open(filename, 'wb') as f:
        f.write(audio.get_wav_data())
    print(f"Saved audio to {filename}")
    return filename

def speak(text: str):
    engine = pyttsx3.init()
    engine.say(text)
    engine.runAndWait()

def main():
    audio_file = record_audio()
    text = transcribe_audio(audio_file)
    print(f"You said: {text}")

    # Simple response logic
    if "hello" in text.lower():
        response = "Hello! How can I help you today?"
    elif "time" in text.lower():
        import datetime
        response = f"The current time is {datetime.datetime.now().strftime('%H:%M')}"
    else:
        response = "I'm not sure I understood that. Try saying 'hello' or 'time'."

    print(f"Assistant: {response}")
    speak(response)

if __name__ == "__main__":
    main()

Run this script, speak into your microphone, and watch your assistant respond. It’s a working prototype you can expand with wake words, LLM integration, or GUIs.

Next Steps: Making It Smarter

This basic assistant is functional, but you can make it powerful by integrating it with an LLM like GPT-4 or a local model like Llama 3. Instead of hardcoded responses, send the transcribed text to an API and return the AI’s reply.

You can also:

• Add a wake word (e.g., “Hey Assistant”) using SpeechRecognition’s keyword detection
• Build a web interface with Streamlit for recording and playback
• Deploy it as a desktop app with PyQt or Tkinter
• Use Faster-Whisper for real-time streaming transcription

Start Building Today

You now have everything you need to build a voice assistant that listens, understands, and speaks-without relying on cloud services. The code above is ready to run, and the architecture is flexible enough to grow with your ideas.

Your challenge: Modify the script to respond to a custom command like “What’s the weather?” by fetching real-time data. Share your version on Dev.to and tag it with #voiceassistant or #whisper. Let’s build the future of human-computer interaction together-one voice command at a time.