How Accurate Is On-Device Whisper Transcription on a Mac? A Real WER Benchmark (2026)

Quick answer: On Ducktate's own benchmark, on-device Whisper (the small.en model) transcribes everyday speech at roughly 12% word error rate with the app's glossary bias on, and about 14% to 15% with it off. That number moves a lot by clip type: numbers and dates score close to 0% WER, ordinary sentences score in the single digits, and out-of-vocabulary brand names and technical jargon are the hard cases, where a custom glossary cuts the error rate roughly in half. This is our own recorded-voice corpus, not a public leaderboard, so treat it as a directional, reproducible number rather than a universal one.

On Ducktate's own transcription benchmark, on-device Whisper (the small.en model, running on Apple's Neural Engine) transcribes everyday speech at roughly 12% word error rate with the app's glossary bias turned on, and closer to 14% to 15% with it off. That is the headline number. The more useful number is underneath it: WER is not one flat figure, it swings from near 0% on plain numbers and sentences to over 20% on brand names and technical jargon the model has never seen, and a custom glossary is what closes most of that gap.

This post is the results table, the methodology, and the caveats behind that number, pulled straight from the eval artifacts Ducktate's own harness writes to disk. Nothing below is a vendor claim about someone else's model. It is what happened when we pointed the app's real transcription code at a fixed set of recorded clips and scored the output against a hand-typed reference.

Results: word error rate by run and by clip

Ducktate's transcription eval (./eval.sh --suite transcription) has been run dozens of times against the small.en model since mid-July 2026, logging every run to eval/history-transcription.jsonl. The numbers below come straight from that log and from the individual run files under eval/runs/transcription/.

Aggregate WER, glossary on vs. off

"Cleaned WER" strips filler words like um and uh from both sides before scoring (Whisper routinely and legitimately drops them, which is not a real transcription error). "Verbatim WER" scores against the literal spoken words, disfluencies included. See Methodology below for why both are reported.

Per-clip breakdown, first fully documented run (2026-07-16)

This is the clearest side-by-side comparison in our logs: the same 5-clip corpus, same model, glossary on vs. completely off, run back to back.

The pattern holds across every later run too: plain speech, numbers, and dates transcribe close to perfectly out of the box. The errors concentrate almost entirely in out-of-vocabulary proper nouns and compound technical terms, and that is exactly where the glossary earns its keep, roughly halving cleaned WER and roughly doubling proper-noun accuracy on the clips that have proper nouns to get wrong.

Methodology

What WER measures. Word Error Rate is the number of word-level substitutions, insertions, and deletions needed to turn the transcript into the reference, divided by the number of words in the reference. Lower is better; 0% is a perfect transcript. It is the standard metric for scoring speech-to-text accuracy, and it is unforgiving: a single wrong proper noun in a short sentence can swing WER by double digits, which is part of why our per-clip numbers above jump around by clip type rather than clustering near one figure.

The corpus. Fixtures/bench/transcription/ in the app repo is real recorded human speech (the founder's own voice, not synthesized), 16kHz mono WAV, each clip paired with a hand-typed reference transcript that is the WER ground truth. Clips are short and deliberately built to isolate a failure mode each: brand and product names, personal names, numbers and dates, natural disfluencies, and compound technical jargon, plus one clip that regression-tests a leading-silence-trim fix in the app's streaming pipeline. A seventh clip of longer natural conversation exists locally but is gitignored (personal content) and is skipped, not scored, in the runs behind this table.

The model. small.en, the Whisper model Ducktate ships by default, transcribing on Apple's Neural Engine, fully on-device. No audio leaves the machine to produce these numbers.

The real pipeline, not a synthetic benchmark. The eval harness decodes each clip and calls the app's actual Transcriber.transcribe(samples:) code path, the same glossary-biasing and language-resolution logic that runs when you dictate, rather than a standalone WhisperKit call. That is a deliberate design choice: it means these numbers are production behavior, not a best-case lab number. --no-glossary reproduces the raw floor for an apples-to-apples comparison.

Two WERs, always. Verbatim WER scores against exactly what was said. Cleaned WER strips bracketed non-speech tokens and disfluencies from both the hypothesis and the reference before scoring, because Whisper models routinely drop filler words as part of normal behavior, not as an error. We report both and treat cleaned WER as the real signal.

Reproduce it yourself: `` ./eval.sh --suite transcription # real path, glossary on ./eval.sh --suite transcription --no-glossary # reproduce the raw floor ``

Honest caveats

This is our own benchmark, and it has real limits worth stating plainly, because the caveats are what make a number trustworthy rather than just favorable.

• - Small corpus. 6 to 7 short clips, a few sentences each, not a large standardized test set like LibriSpeech. A handful of clips is enough to see a clear pattern, not enough to bound accuracy tightly to a decimal point.
• - One voice. Every clip is the founder's own recorded speech. We have not yet measured WER across different accents, speaking speeds, microphones, or background noise conditions, all of which are known to move Whisper's accuracy meaningfully.
• - Real run-to-run variance. Running the identical model against the identical corpus did not produce the identical number. Across 35 valid glossary-on runs in our log, cleaned WER ranged from 6.3% to 13.1%. That spread is on-device decoding variance, not a data-entry error, and we are reporting it rather than picking the most flattering run.
• - We excluded broken runs, and are telling you so. A stretch of runs in our log show cleaned WER as high as 81% with only 25% to 50% of clips scoring at all (passRate below 1). Those came from a period where the app itself was mid-debugging and clips were failing to transcribe, not a measurement of real transcription quality, so we left them out of the table above. We are naming that exclusion explicitly rather than quietly dropping the rows.
• - Thin glossary-off sample. Only 2 logged runs exist with the glossary fully disabled, versus 35-plus with it on. The direction of the effect (glossary lowers WER, especially on proper nouns) is consistent and large, but the off-condition sample size is small.
• - One model size. These numbers are all small.en. We have separately observed that a larger model (distil-large-v3) does not fix the out-of-vocabulary failure mode that drives most of the error here, but we have not run the full WER suite against every model tier Ducktate ships.
• - Your results will differ. WER is sensitive to accent, audio quality, background noise, and how much domain-specific or invented vocabulary you use. Treat these numbers as evidence about the shape of the problem (plain speech is easy, proper nouns and jargon are hard, a glossary helps) rather than a guarantee of a specific percentage on your own voice.

Why this matters for dictation, not just transcripts

Raw transcription accuracy is the floor under everything else a dictation app does. Talking to an AI is faster than typing to one only holds up if what gets transcribed is actually what you said, and a good Mac dictation app has to get proper nouns and jargon right, since those are exactly the words a wrong guess makes you stop and manually fix. That is the whole argument for a glossary: it is not a nice-to-have, it is roughly a 2x accuracy difference on the words that cost you the most time to correct by hand.

If you want to see how much of your own speech is filler words and disfluencies, the kind that verbatim WER counts and cleaned WER forgives, try the free filler word counter: paste or record a sample and see your own um/uh rate instead of someone else's average.