AI Speech Recognition in Hospitals: the Who-Said-What Problem

An admission interview in internal medicine, half past ten. The patient is 84; her daughter sits beside her and helps where the hearing aid falls short. Asked about heart complaints, the daughter answers: "She never had chest pain." The patient nods and murmurs something about her knee. A monitor runs in the next room; out in the hallway, someone is calling for physiotherapy.

For a human, the scene is unremarkable. For a speech recognition system, it is the hardest task there is: multiple speakers, overlaps, background noise, distance to the microphone — and a statement whose clinical meaning depends on who made it. "No chest pain," reported by the patient herself, is a history item. The same words, wrongly attributed to the physician, become a finding.

Dictation is solved. Conversation is not.

The numbers have been stable for years and are rarely shown in vendor presentations. A systematic review of clinical speech recognition found word error rates below 9 percent for controlled dictation — and above 50 percent in conversational multi-speaker scenarios [1]. In the international DIHARD III challenge, which measures speaker attribution on difficult real-world audio, the best system achieved a diarization error rate of just under 17 percent [2] — roughly speaking: every sixth second of speech is attributed to the wrong person or missed. And the error rate grows with the number of participants: in benchmark measurements it nearly doubles when four people speak in the room instead of two [3]. That is precisely the reality of an admission interview with family members present.

Add to this what makes hospitals acoustically distinctive. In meeting corpora, around 12 percent of speaking time is overlapped; for short confirmations ("mhm"), the figure is roughly 70 percent [4]. Masks dampen exactly the frequency bands from which systems distinguish voices [5]. And a 2026 stress test of a complete pipeline of speech recognition and language model showed: with background chatter, the share of clinically unsafe outputs rose from 13.6 to 91.5 percent [6].

Why this error class matters clinically

Research on AI scribes has now measured the consequences well. An analysis of five commercial platforms found an average of just under 14 transcript errors per conversation; around three quarters of all errors in the finished notes were omissions [7]. A paper in npj Digital Medicine names speaker confusion as an error class of its own: patient statements end up in the physician's mouth — and vice versa [8]. Those who evaluate such systems today usually ask about hallucinations. The more frequent errors are the missing and the swapped.

Two ways to attack the problem

The classic route improves voice recognition itself: better acoustic signatures, better segmentation. In real rooms it runs into limits that are physical — distance, reverberation, similar voices, overlap.

The second route uses content. Whoever says "Where does it hurt?" is very probably not the patient. A Google research team showed as early as 2019, on physician-patient conversations, what this additional information is worth: the word-level speaker error rate fell from 15.8 to 2.2 percent when the system also used linguistic signals [9]. More recent work using large language models as a correction layer halves the speaker errors once again [10].

At aiomics, the conversation documentation for the admission interview is in production — deliberately cut narrowly to exactly this scenario. Speaker attribution does not rely on the sound of the voice alone but draws on the content of what is said — whoever asks about complaints speaks recognizably differently from whoever describes them. Every transcript segment carries a speaker attribution, a timestamp, and a confidence indication; segments whose attribution remains uncertain are flagged rather than silently adopted. Negations — "no allergies", "never chest pain" — are displayed with the original quote, because a swapped negation inverts the meaning; exactly this error class is measured separately. And because room acoustics are often the limiting factor, the microphone situation is part of the rollout decision: close-talking microphones save several error points over a room microphone in benchmarks, and multi-channel processing lowered the word error rate by almost 20 points in the CHiME challenge [11] — and where the rooms require it, we provide multi-microphone devices.

The result is a draft with timestamps, confidence indications, and sentence-level references into the transcript. It is the physician who assesses it. Processing runs exclusively on self-hosted infrastructure in the EU; audio is deleted by default after 72 hours, with proof.

The regulatory line is foreseeable

In Sweden, the Medical Products Agency began inspecting vendors of AI conversation documentation in 2026 and, in the case of the Swedish vendor Tandem Health, provisionally recorded — according to Tandem's own summary — that an AI scribe whose outputs become the basis of care decisions is at minimum a Class IIa medical device [12]. Tandem has since, according to the company, achieved the corresponding certifications for its coding assistant and scribe. In Germany, no comparable regulatory position exists so far; but the governing Rule 11 of the MDR and the European guidance MDCG 2019-11 point in the same direction [13]. It is to be expected that this classification line will also reach the German market — and individual vendors are already preparing for it. We are among them: for our conversation documentation, Class IIa certification is in preparation.

What to measure any system against

1. Ask to see speaker attribution in a multi-person scenario with family members present; the two-person conversation of the demo is the easiest case.
2. Ask how the system handles uncertain attributions: does it flag them, or does it decide silently?
3. Ask about the handling of negations — the error class with the highest damage potential.
4. Ask where audio is processed and when it is verifiably deleted.
5. Ask about the medical device classification and its justification — not just the outcome.

If you want to play through the who-said-what problem in your own facility on real scenarios, write to us. Ongoing analysis of clinical AI appears in our newsletter Visite — an English edition, Grand Rounds, has a waitlist.

Sources

1. Systematische Übersichtsarbeit zu klinischer Spracherkennung. BMC Medical Informatics and Decision Making. 2025;25:236.
2. DIHARD III Challenge — offizielle Ergebnisse. dihardchallenge.github.io/dihard3/results.
3. NVIDIA Sortformer Modellkarte, CALLHOME-Benchmarks (2–4 Sprecher). huggingface.co/nvidia/diar_sortformer_4spk-v1.
4. Çetin Ö, Shriberg E. Analysis of overlaps in meetings. Interspeech 2006.
5. Studien zu Maskeneffekten auf Sprachsignal und Sprechererkennung. PMC9003118; PMC10225750.
6. Stress testing ASR-LLM clinical documentation pipelines under acoustic noise. arXiv:2606.05909, 2026.
7. Fehleranalyse über fünf Ambient-Dokumentationsplattformen: Transkript- und Notizfehler. Mayo Clinic Proceedings: Digital Health, 2025.
8. Sprecherverwechslung als dokumentierte Fehlerklasse in KI-Scribe-Dokumentation. npj Digital Medicine. 2025;8:569.
9. El Shafey L, Soltau H, Shafran I. Joint speech recognition and speaker diarization via sequence transduction. Interspeech 2019.
10. Wang Q et al. DiarizationLM. arXiv:2401.03506, 2024.
11. CHiME-6 Challenge. Watanabe S et al. arXiv:2004.09249, 2020; pyannote-Benchmarks, github.com/pyannote/pyannote-audio.
12. Tandem Health. Zusammenfassung der vorläufigen Inspektionsergebnisse der schwedischen Arzneimittelbehörde (Dnr 6.6.3-2025-109474). Unternehmensmitteilung, Mai 2026; Läkartidningen, 04.03.2026.
13. MDCG 2019-11 Rev. 1 (17.06.2025); Verordnung (EU) 2017/745, Anhang VIII, Regel 11.

Sources retrieved in July 2026.