Whispering in Norwegian: Navigating Orthographic and Dialectic Challenges

Per E Kummervold, Javier de la Rosa, Freddy Wetjen, Rolv-Arild Braaten, Per Erik Solberg · pdf · 2024-02-02 Reader 🖥️

Automatic Speech Recognition (ASR)Data Sovereignty Evals Open-Weight Models

NB-Whisper — OpenAI's Whisper fine-tuned by the National Library of Norway on roughly 28,000 hours of curated, iteratively-cleaned Norwegian audio-text data from four sources (NRK subtitles, audiobooks, the NST corpus, and the Stortinget speech corpus) — substantially reduces word-error-rate on Norwegian Bokmål and Nynorsk transcription (e.g. 10.4→6.6 on Fleurs, 6.8→2.2 on NST, 30.0→12.6 on Common Voice Nynorsk) by explicitly targeting the dual written standards and pervasive dialectal/orthographic variation that generic Whisper handles poorly, though an inherent ~1-2% WER floor from legitimate Norwegian spelling variation and unresolved hallucination/streaming limitations mean the problem is substantially advanced rather than solved.

theoretical · expository — An empirical NLP systems paper: it reports what was built and measured (a fine-tuned ASR model and its word-error-rate results) rather than prescribing action. Structured like a standard ML conference paper (intro, related linguistic background, architecture, dataset, evaluation, results, limitations, future work) with argumentative passages defending specific design choices (why not teacher-student pseudo-labeling, why light normalization) embedded in an otherwise expository report.

Structure

Part	Role
Introduction	Motivates the gap: Whisper is state-of-the-art but its weak-supervision approach, tuned mostly on English/high-resource data, underperforms on Norwegian's dialects and dual orthography.
Language Variability and ASR Challenges	Establishes the linguistic facts an ASR system for Norwegian must handle: two co-official written standards (Bokmål, Nynorsk), intra-standard orthographic variation, and dialects with no official spoken norm, evidenced by e.g. 38 recorded pronunciations of 'hvem'.
Model Architecture and Training	Describes retaining Whisper's encoder-decoder Transformer architecture while modifying hyperparameters (batch size 1024, increased BPE dropout, activation dropout instead of stochastic depth) and running a two-phase training schedule (200k steps, then 50k steps on a cleaned subset).
Dataset	Details the four training sources (NRK subtitles, audiobooks, NST, Stortinget Speech Corpus; ~22,184h stage 1 / 6,078h stage 2), the alignment techniques (subtitle timestamps, NB-Wav2Vec2 forced alignment), English-translation augmentation, and a multi-criteria automated cleaning pipeline (fuzzy first/last-word matching, insertion/omission n-gram detection, NER-based filtering).
Experimental Setup and Evaluation	Defines the three Norwegian test sets used (Fleurs, NST, Common Voice Nynorsk) and the WER methodology (JiWER, lowercased, punctuation-stripped, non-normalized for cross-dataset comparability).
Results and Discussion	Reports WER across five model sizes (Tiny→Large) and three datasets, showing NB-Whisper beats OpenAI Whisper at every size, with the largest relative gains on Nynorsk and at smaller model sizes.
Challenges and Limitations	Acknowledges that hallucination and long-text transcription were only evaluated qualitatively (no suitable Norwegian benchmark exists), and that the autoregressive, 30-second-chunk architecture is a poor fit for real-time/live transcription.
Future Work	Proposes using an LLM to harmonize orthographic variation in the training corpus to a consistent style, and re-cleaning the dataset with improved tooling.
Conclusion	Restates the achievement and frames NB-Whisper's methodology as transferable to other dialect-rich, dual-standard languages.

Key terms

Word Error Rate (WER) Automatic Speech Recognition (ASR)
The paper's central metric, computed with the JiWER package after lowercasing and stripping punctuation, deliberately reported non-normalized so scores are directly comparable across the three test datasets; the authors note that a stronger normalization (e.g. spelling out numbers, as in a companion Wav2Vec2 paper) would have produced lower, but less comparable, scores.
Bokmål / Nynorsk
Two co-official written standards that any Norwegian ASR output must consistently target regardless of the speaker's dialect — the paper treats 'correct transcription' as convergence on one of these written norms, not a phonetic rendering of speech.
Dialect
Spoken variation that is decoupled from the written standard: there is no official spoken norm, dialects are used even in parliament and news broadcasts, and a single interrogative pronoun ('hvem'/'kven') has 38 recorded pronunciations in the Nordic Dialect Corpus — dialect is presented as an input-side nuisance variable the model must normalize away, not something to preserve in output.
Weak supervision (Whisper-style pretraining)
Pretraining on a large, loosely-aligned corpus of subtitles (as opposed to a verbatim phoneme-to-grapheme dataset) that lets the model learn punctuation, capitalization, and natural written-language phrasing in one step — inherited from Whisper and extended here with Norwegian-specific weakly-aligned sources (NRK subtitles, Stortinget proceedings).
Dataset cleaning via own-model filtering
A specific two-stage procedure: train on the full noisy dataset, run the resulting model's own predictions against the training targets, and use four criteria (fuzzy first/last-word match, detection of target insertions absent from any prediction, detection of omissions present in predictions but absent from target, NER-based filtering) to delete bad snippets — explicitly not a teacher-student pseudo-labeling approach, since the model's outputs are used only to filter, never as new training targets.
Orthographic variation
The existence of multiple, simultaneously 'correct' spellings for the same word within a single written standard (e.g. Bokmål 'lua'/'luen', Nynorsk infinitives '-e'/'-a') — treated as an inherent ceiling on achievable WER, not a model deficiency: even professionally narrated, force-aligned audiobooks only reach ~1-2% WER because of this.

Propositions

explicitNB-Whisper Large reduces Norwegian Bokmål WER on the Fleurs dataset from 10.4 (OpenAI Whisper Large v3) to 6.6.conf: high
explicitNB-Whisper Large reduces Norwegian Bokmål WER on the NST dataset from 6.8 (OpenAI Whisper Large v3) to 2.2.conf: high
explicitOn Common Voice Nynorsk, NB-Whisper Large achieves 12.6% WER versus 30.0% for OpenAI Whisper Large v3 — a proportionally larger improvement than on Bokmål datasets.conf: high
explicitAt the Tiny model size, NB-Whisper achieves 28.0% WER on Common Voice Nynorsk while OpenAI Whisper Tiny exceeds 100% WER (more insertions/errors than reference words) — i.e., stock small Whisper models are close to unusable for Nynorsk.conf: high
explicitEven under near-ideal alignment conditions (professionally narrated audiobooks, forced-aligned), the achievable WER floor is approximately 1-2%, because Norwegian permits multiple simultaneously correct spellings of the same word.conf: high
explicitNRK subtitles are non-verbatim relative to spoken audio, giving a baseline word-to-word WER of around 30% against them independent of any ASR error.conf: high
explicitThe final training corpus totals roughly 22,184 hours (stage 1) and 6,078 hours (stage 2) drawn from five sub-sources: NRK subtitles, NRK no-caption audio, audiobooks, the NST dataset, and the Stortinget Speech Corpus.conf: high
explicitdevelopedASR performance scales strongly and roughly monotonically with model size for Norwegian (Tiny through Large), for both OpenAI Whisper and NB-Whisper, but the absolute and relative gap between the two narrows as size increases.conf: high
extends ←Generative AI in the Real World: Local Voice AI with Pete Warden · p11 — Adds a contrasting data point to the 'ASR performance scales with model size' claim: a general multimodal model (Qwen ~1.7B) reportedly outperforming the dedicated Whisper architecture on some languages, complicating a pure bigger-specialized-model-always-wins story.
implicitThe own-model-filtering, two-stage training approach used here (predictions used only to filter/delete bad snippets, never as new training targets) is implied to be preferable to a teacher-student pseudo-labeling approach for reducing hallucinations, though no direct ablation against teacher-student retraining is reported within the paper.conf: medium
explicitNB-Whisper's architecture (autoregressive decoder, 30-second audio chunks) is not well-suited to real-time/live transcription, and would benefit from an alternative architecture for that use case.conf: high

Arguments

NB-Whisper achieves substantially lower word-error-rate than OpenAI's stock Whisper for both Norwegian Bokmål and Nynorsk transcription. strength: medium

Explicit premises:

Fine-tuning was performed on ~28,000 hours of Norwegian-specific audio-text data from four curated sources.
Hyperparameters (batch size, BPE dropout, activation dropout, weight decay) were tuned specifically for this corpus.
A two-stage training regime with automated, multi-criteria data cleaning between stages was used to reduce hallucinations.
Evaluation was run on three independent Norwegian test sets (Fleurs, NST, Common Voice Nynorsk) using a fixed WER methodology (JiWER, lowercased, punctuation-stripped).

Implicit premises:

The chosen test sets (Fleurs, NST, Common Voice) are representative proxies for real-world Norwegian ASR use, including realistic dialect diversity and both written standards — despite Fleurs and NST both being read/scripted speech rather than spontaneous conversation.
WER computed on lowercased, punctuation-stripped text is a meaningful proxy for practical transcript usability, even though it discards capitalization and punctuation quality that Whisper-style models are specifically praised elsewhere in the paper for producing.
Confirming no speaker overlap between train and test splits is sufficient to rule out leakage-inflated scores; no other overlap check (e.g. content/topic overlap between NRK training data and NST/Fleurs) is claimed.
An improvement over OpenAI Whisper's numbers constitutes meaningful progress on 'the' Norwegian ASR problem, even though the paper itself documents an inherent ~1-2% orthographic-variation floor and unresolved hallucination/streaming limitations.

Inferences:

B
e
f
o
r
e
/
a
f
t
e
r
c
o
m
p
a
r
a
t
i
v
e
b
e
n
c
h
m
a
r
k
i
n
g
a
t
t
r
i
b
u
t
e
s
t
h
e
e
n
t
i
r
e
o
b
s
e
r
v
e
d
W
E
R
d
e
l
t
a
t
o
t
h
e
N
B
-
W
h
i
s
p
e
r
t
r
a
i
n
i
n
g
a
p
p
r
o
a
c
h
(
a
r
c
h
i
t
e
c
t
u
r
e
h
e
l
d
c
o
n
s
t
a
n
t
)
,
a
n
d
a
s
c
a
l
i
n
g
t
r
e
n
d
a
c
r
o
s
s
f
i
v
e
m
o
d
e
l
s
i
z
e
s
i
s
u
s
e
d
t
o
g
e
n
e
r
a
l
i
z
e
c
l
a
i
m
s
a
b
o
u
t
t
h
e
r
e
l
a
t
i
o
n
s
h
i
p
b
e
t
w
e
e
n
m
o
d
e
l
s
i
z
e
a
n
d
N
o
r
w
e
g
i
a
n
A
S
R
a
c
c
u
r
a
c
y
.

Norwegian's dual written standards plus pervasive, spoken-norm-free dialectal variation are the primary reason generic Whisper underperforms on Norwegian, and are what NB-Whisper is specifically engineered to address. strength: medium