Show HN: Live speech-to-text on an ESP32-P4 microcontroller, no cloud

Live English speech recognition on the ESP32-P4. Open vocabulary, streaming, faster than real time, on the microcontroller alone: the board's microphone in, the words on its screen as you speak. No cloud, no neural accelerator; the two RISC-V cores and their PIE SIMD instructions.

The model is Kroko (Banafo), a 65.7M-parameter streaming Zipformer2 transducer, with its weights brought from 71 MB to 26.65 MB in 2 and 4 bits. The weights are on Hugging Face: BoumedineBillal/esp32-pie-asr-kroko-en.

Performance

| Accuracy | LibriSpeech test-clean 4.34 % WER, test-other 10.26 % (beam search, 4 hypotheses) |
| Speed | 0.70× real time: each 1.28 s of audio in 0.90 s, both cores at 400 MHz |
| Delay | a word is final about 2 s after it is spoken (median 1.9 to 2.3 s, 95 % within 4.1 s) |
| Weights | 26.65 MB, read from a microSD card at boot in 2.9 s |
| Memory | weights and state in the 32 MB PSRAM; 121 KB of internal RAM left free (104 KB with the demos) |
| Bit-exact | the chip computes exactly what the engine's PC build computes: the PC's WERs are the chip's |
| Phrases | words and sentences to listen for (hotwords), switched on the screen, no retraining |
| Voice control | six demos driven by spoken orders through word rules, no language model |

The firmware, firmwares/voice_control, opens on live transcription, then the voice-control demos. The transcriber's numbers were measured with firmwares/live_transcriber, the transcriber alone on the same engine, where the engine's board tests are.

Comparison

| Parameters | Weights | Runs on | Streaming | test-clean | test-other | Speed |
|---|---|---|---|---|---|---|
| esp32-pie-asr | 65.7M | 2/4-bit, 26.65 MB | ESP32-P4 (2 × 400 MHz RISC-V) | yes | 4.34 % | 10.26 % | 0.70× real time |
| Kroko as released (sherpa-onnx) | 65.7M | int8, 71 MB | PC | yes | 3.94 % | 9.17 % | |
| Oído, streaming, with LM | 13M | int8 | ESP32-S3 (240 MHz Xtensa) | yes | 4.86 % | 11.03 % | final text 4.1 to 11.1 s after clips of 3.6 to 10.6 s |
| Oído, whole utterance | 13M | int8 | ESP32-S3 | no | 3.70 % | 8.23 % | 1.97× real time (one core) |
| Moonshine Tiny | 27.1M | float | CPU | no | 4.52 % | 11.71 % | |
| Whisper tiny.en | 37.8M | float | CPU | no | 5.66 % | 15.45 % | |
| ESP-SR MultiNet | ESP32-S3 | a fixed list of up to 200 commands, not open vocabulary |

esp32-pie-asr and Kroko as released: measured here on all 2620 / 2939 clips with Whisper's English text normalizer (Kroko with sherpa-onnx 1.13.8, the same beam width, 2 s of silence after each clip so its last chunk is decoded). The others: as their authors publish them.

Where the accuracy goes, test-clean:

| greedy | beam 4 |
|---|---|
| Kroko as released, int8 ONNX | 4.03 % | 3.94 % |
| 2/4-bit weights, float activations (PyTorch simulation) | 4.53 % | |
| 2/4-bit weights, integer activations: the chip | 4.52 % | 4.34 % |

The 2/4-bit weights cost about half a point; the integer arithmetic on the chip costs nothing.

Phrases

A text file on the SD card lists words and sentences to listen for; a word the model spells differently gets its spoken forms (ESP32 = ESP thirty two). Each one is switched on or off on the screen.

On 140 text-to-speech clips (35 sentences, 4 voices) with the default list:

| the list's words found | its sentences found | false alarms (40 clips without them) | WER |
|---|---|---|---|
| phrases off | 0 of 60 | 62 of 72 | 0 | 19.5 % |
| phrases on | 46 of 60 | 70 of 72 | 0 | 3.4 % |

Voice control

The firmware's second page has six demos in which spoken orders drive a device simulated on the screen: a robot vacuum, a smart home, a car cabin, a factory line, a kitchen, a presenter. No language model: each demo is a table of word rules (an order's verb, then its target; numbers, rooms, "not the bedroom"), and the screen shows which words fired each action. Each demo has its own phrase list.

On 188 text-to-speech clips per demo (25 orders and 22 near misses, ordinary sentences with the demo's words, 4 voices):

| orders exact, no list | with its list | false actions (88 near misses) |
|---|---|---|
| robot vacuum | 92 % | 96 % | 0 |
| smart home | 93 % | 97 % | 0 |
| car cabin | 93 % | 97 % | 0 |
| factory line | 86 % | 94 % | 0 |
| kitchen | 97 % | 98 % | 0 |
| presenter | 97 % | 99 % | 0 |

The lists were chosen on these clips; every miss is a word heard wrong. The board gives the PC's actions on every clip.

From the end of an order to its action: median 1.84 s on the vacuum's page, 1.97 s on the factory's, at most 2.57 s. With a demo's page drawing, the engine stays under real time (at most 0.81×).

Hardware

Hardware: an ESP32-P4 of chip revision v3 (ESP32-P4X) with 32 MB PSRAM, a microphone, a screen, a microSD card. Made and measured on Espressif's ESP32-P4X-Function-EV-Board (ES8311 microphone, 1024 × 600 MIPI-DSI panel).

The weights (26.65 MB) do not fit the board's 16 MB flash, so they live on a microSD card: at boot the firmware reads them from the card into the 32 MB PSRAM (2.9 s) and runs from there.

• Put the weights on a FAT-formatted microSD card, in its top folder:

git clone https://github.com/BoumedineBillal/esp32-pie-asr
cd esp32-pie-asr
python models/download.py --card # downloads, checks, copies

• Put the card in the board's microSD slot.
• Flash the firmware, from the browser (Chrome or Edge, no tools to install): boumedinebillal.github.io/esp32-pie-asr, or with ESP-IDF v6.1:

cd firmwares/voice_control
idf.py set-target esp32p4
idf.py -p PORT flash

At the first boot the firmware also writes its phrase lists on the card: phrases.txt for the transcriber and one per demo (vacuum_phrases.txt, smarthome_phrases.txt, ...). To change a list, edit its file on a PC: one word or sentence per line, shown = spoken | spoken for a word the model spells differently, # for a note; the screen shows up to 10 words and 10 sentences.

firmwares/live_transcriber (the transcriber alone) builds and flashes the same way. The debug build adds a link to a PC over the board's USB High-Speed port (commands, the log, the words, screenshots) for the board tests (debug/tests/run_all.py, Windows):

idf.py -B build_debug -D SDKCONFIG=build_debug/sdkconfig -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.debug" -p PORT flash

How the model fits and runs

How the model fits and runs, in docs/design.md:

• 2/4-bit weights in groups of 128, with a Hadamard rotation and one norm per group; 2 or 4 bits chosen per matrix by its sensitivity, without training.
• Integer arithmetic: int8 into the products, int16 between them, int8 attention caches, activations by table.
• PIE SIMD kernels written in assembly, each with a C reference that the host build runs; the products split over both cores.
• Bit-exact with the PC: deterministic math on both sides, checked on every test clip (the encoder's output CRC and the text) and on every kernel.
• Streaming in 1.28 s chunks with fixed-size caches; a beam search with phrase biasing; the transcript's word and line rules.
• Voice control by word rules, tested on the PC; on its pages a pause commits the search's best hypothesis, so an order acts about 1.9 s after its end instead of 3.1 s.

cmake -S pieasr/host -B build/host -DPIEASR_WEIGHTS=$PWD/models/kroko-en-q24-v1.bin
cmake --build build/host && ctest --test-dir build/host # unit tests and the 8 golden clips
python tools/eval/get_librispeech.py --set test-clean # 2620 clips; --set test-other: 2939
bash tools/eval/run_wer.sh data/test-clean int+beam4 # the chip's configuration, on the PC
bash pieasr/tests/target/run.sh # every PIE kernel against its C reference, on the board

• Support for the ESP32-S31, which has the same PIE 2.2 SIMD instructions as the ESP32-P4X.
• Arabic: a model for Modern Standard Arabic.
• Kroko by Banafo: the model (community license, CC-BY-SA).
• Cactus Quants by Cactus Compute: the weight format (Hadamard-rotated groups of 128, Lloyd-Max levels).
• LibriSpeech (CC BY 4.0): the 8 test clips in pieasr/tests/data and the WER sets.
• Piper and its voices: the text-to-speech test clips (made by the tests' scripts, not in the repo).
• Espressif: ESP-IDF, and the ES8311 and EK79007 register values (Apache-2.0).
• Inter (SIL OFL 1.1) and stb_truetype (public domain or MIT).

The code: Apache-2.0 (LICENSE). The weights: CC-BY-SA 4.0, as Kroko's.

Boumedine Billal: GitHub · Hugging Face · LinkedIn · bou************@gmail.com

@misc{boumedine2026esp32pieasr,
author = {Boumedine Billal},
title = {esp32-pie-asr: streaming speech recognition on the ESP32-P4},
year = {2026},
url = {https://github.com/BoumedineBillal/esp32-pie-asr}
}