Speech recognition

How ASR works, conceptually

Modern speech recognition is a neural network that maps audio features to text tokens. The pipeline, simplified:

1. Feature extraction. Raw samples become a spectrogram — energy across frequency bands over time. This is what the model actually consumes.
2. Acoustic modelling. A network (these days usually a transformer or conformer) maps spectrogram frames to a probability distribution over tokens.
3. Decoding. A search finds the most likely token sequence, often with a language model biasing toward plausible word sequences.

Older systems had explicit phoneme dictionaries and hand-built pronunciation rules. Modern end-to-end models learn the mapping directly from audio-text pairs, which is why they handle accents and informal speech dramatically better.

Streaming vs batch

A batch model sees an entire recording and produces one transcript. Maximum accuracy, because it has full context in both directions — but you wait for the recording to end.

A streaming model emits text as audio arrives. Necessary for dictation, but it must commit to words before hearing what follows, which occasionally produces a wrong guess that later context would have fixed.

The reconciliation is volatile results. The model emits its best current guess, marked as provisional; later it either confirms that text as final or replaces it. You'll literally watch words change on screen as you keep talking — that's not a bug, it's the model incorporating new context.

private func apply(text: String, isFinal: Bool) {
    if isFinal {
        finalizedText += text     // locked in, won't change
        volatileText = ""
    } else {
        volatileText = text       // replaced wholesale next update
    }
}

Note the asymmetry: finalized text accumulates, volatile text is replaced. Getting this backwards produces either duplicated words or a transcript that keeps erasing itself — and it's the most common bug when writing this integration by hand.

Apple's SpeechAnalyzer architecture

Introduced at WWDC25 for macOS 26, SpeechAnalyzer replaced the older SFSpeechRecognizer. It's meaningfully better: fully on-device, genuinely streaming, and no longer subject to the arbitrary duration limits that made the old API frustrating for dictation.

The design is modular. An analyzer coordinates one or more modules; SpeechTranscriber is the speech-to-text one.

let transcriber = SpeechTranscriber(
    locale: resolved,
    transcriptionOptions: [],
    reportingOptions: [.volatileResults],
    attributeOptions: []
)

let analyzer = SpeechAnalyzer(modules: [transcriber])
try await analyzer.start(inputSequence: stream)

for try await result in transcriber.results { … }

Four things worth knowing about the surrounding lifecycle:

Model assets download on demand, per language. AssetInventory.assetInstallationRequest(supporting:) returns nil if the model is already installed, or a request you must download. Do this at launch, not on the first hotkey press — otherwise the user's first dictation stalls behind a download.

Locales must be resolved, not assumed. SpeechTranscriber.supportedLocale(equivalentTo:) maps Locale.current onto a locale the model actually ships. Passing en_GB when the model expects en_US fails quietly.

Ask the analyzer for its preferred audio format via bestAvailableAudioFormat(compatibleWith:), rather than hardcoding 16 kHz. It's async but doesn't throw.

Finish properly. finalizeAndFinishThroughEndOfInput() flushes audio still sitting in the model's internal buffer. Skip it and you lose the last word or two of every utterance — a subtle bug that makes an app feel broken without users being able to say why.

Where local still loses

Be honest about this. Cloud models are larger, and it shows in three places: non-English languages, code-switching mid-sentence, and unusual proper nouns. Apple's on-device model is very good at conversational English and noticeably weaker outside it.

The strategic answer isn't to close that gap — it's to not compete there. English-first, privacy-absolute, offline, free. That's a coherent product. "Slightly worse at 100 languages" is not.