Whisper backend shootout: faster-whisper vs HF Transformers vs whisper.cpp (2026)

TL;DR — On an RTX 4070 Ti with telephony-optimised audio preprocessing, faster-whisper with large-v3-turbo reaches ~56× real-time — the fastest overall. whisper.cpp CUDA is close (~40×) at a fraction of the VRAM. Batched HF Transformers (SDPA, batch 24) is competitive on the turbo model. On CPU, whisper.cpp surprisingly matches GPU speeds for short files; faster-whisper int8 stays practical at ~5×.

Part of the Danish voice-log project — this is the backend benchmark behind that pipeline’s ASR choice. See also running Whisper locally.

Setup

Audio test set

Three real Danish phone-call recordings from a Danish voice-log corpus (filenames anonymised to File 1–3):

No reference transcriptions exist — quality is evaluated via consensus WER (pairwise agreement across models) and relative WER (distance to the consensus winner).

Backends under test

Results: speed (RTF — lower is better)

RTF = transcription time ÷ audio duration. RTF 0.03 = 33× real-time.

Results: VRAM usage (GPU runs)

Results: quality (consensus WER)

Consensus WER measures how much each model’s output diverges from the group mean. Lower = more similar to what all models agree on.

Quality winner: faster-whisper / large-v3-turbo / CPU

Side-by-side: where the backends differed

The test audio is a private call, so rather than reproduce it, here’s a neutral Danish sentence standing in for the kind of conversational speech tested — plus the divergence patterns that actually showed up across backends on the 86-second file:

“Lad os tage mødet på torsdag, så gennemgår vi tallene sammen.” (illustrative)

• Dropped openings. faster-whisper large-v3 (CUDA and CPU) sometimes started a few words in, omitting an opening sentence that the turbo models and whisper.cpp captured.
• Proper-noun disagreement. The opening name was transcribed two different ways across backends — the classic low-resource-name problem, where each model guesses a different plausible spelling.
• Garbled noun phrases on noisy segments. whisper.cpp large-v3 turned a clear noun phrase into a near-homophone that the turbo models got right.
• Rare hallucinated fragments. faster-whisper large-v3 occasionally emitted a short nonsense phrase on a noisy stretch where the other backends stayed clean.
• Minor spelling slips. The turbo CPU run produced the odd vowel typo not seen on GPU.

Net: the turbo models were the most consistently coherent; full large-v3 was strong but more prone to dropped openings and the occasional hallucination on this noisy telephone audio.

Key findings

• Fastest GPU: faster-whisper / large-v3-turbo / CUDA at avg RTF 0.018 (56× real-time).
• Turbo speedup (faster-whisper GPU): large-v3-turbo is 2.4× faster than large-v3.
• Fastest CPU: whisper.cpp / large-v3-turbo / CPU at avg RTF 0.026.
• GPU vs CPU (faster-whisper turbo): GPU is 11.4× faster than CPU int8.
• Lowest VRAM: HF Transformers (b24, SDPA) / large-v3-turbo / CUDA at 3884 MB.

Recommendations

Methodology notes

• Each backend/model/device combination runs as a separate process to guarantee full GPU memory reclamation between runs.
• faster-whisper GPU uses compute_type=float16; CPU uses int8.
• HF Transformers uses attn_implementation=sdpa (PyTorch SDPA / FlashAttention-equivalent, no extra install).
• whisper.cpp uses CUDA automatically when compiled with GGML_CUDA=ON — no flag needed; CPU runs use -t 8 (system thread count).
• All audio is pre-processed with ffmpeg before transcription: highpass=f=200,lowpass=f=3400,afftdn=nr=10:nf=-25,dynaudnorm (telephone band-pass + light denoise + loudness normalisation).
• VRAM is measured via torch.cuda.max_memory_allocated(); not available for the whisper.cpp subprocess runs.
• All audio is Danish conversational phone-call speech — results may vary on other languages or audio conditions.
• Consensus WER is computed pairwise across all models; the model with the lowest mean pairwise WER is the “consensus winner”.