Local AI on a PC: what actually works (Whisper, translation, LLMs)
Local AI is the most overpromised topic in dev tooling. "Run GPT on your laptop" - sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second.
I've been running local models in production-ish settings for a while (speech recognition, translation, TTS, document analysis). Here is what actually works on consumer hardware, and what doesn't.
Speech recognition: faster-whisper
The best value-per-watt local model I use. CTranslate2 backend, int8 quantization.
mediumint8on an RTX 2070: ~4.4 GB of VRAM, a 30-second fragment in 0.36-1.62 s depending on speech density;- CPU works too, several times slower - fine for batch transcription, too slow for real-time subtitles;
- hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it.
Translation
- Helsinki-NLP OPUS models: ~74 MB, fast, quality is decent for short, well-formed sentences - exactly what subtitles are;
- NLLB / M2M100: better quality, but check the licence. NLLB is CC BY-NC - fine for personal use, not for a commercial product. I learned that the hard way and swapped it;
- for anything legal or medical, local NMT is not good enough and you should say so rather than ship it.
TTS
- Piper: tiny, fast, runs on CPU, sounds robotic but intelligible - good for notifications and accessibility;
- XTTS: much better quality, needs a GPU and a licence read.
LLMs
A 7-8B model in Q4 quantization takes 8-12 GB of VRAM and produces 20-40 tokens/sec on a decent GPU. That is enough for:
- summarising long documents;
- extracting fields from PDFs and invoices;
- classification and routing;
- "explain this error" against your own logs.
It is not enough to replace a frontier model for open-ended reasoning - and if you're paying per token, the API is cheaper until you cross a volume threshold.
The decision that actually matters
Local vs API is an economics and privacy decision, not a religious one:
- API is cheaper at the start - dozens of operations a month don't justify hardware;
- local is cheaper at volume - thousands of documents a month amortise the GPU in months;
- local is mandatory for sensitive data - medical, personal, trade secrets;
- hybrid is often best - sensitive data locally, human-facing drafts through an API.
Start with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes.
Longer write-up with the hardware table, model sizes and real benchmarks: Local AI on a PC: what actually works. A real product built this way: Stream-Subs.
Comments
No comments yet. Start the discussion.