Shadow AI in the Department: Lead It, Don't Ban It

While reading over the letters, the chief physician notices that one final-year student's drafts have been sounding remarkably polished for a few weeks. A friendly question gets a friendly answer: "I run it through ChatGPT to smooth it out." On the private phone, with the private account. Around the table, two residents nod. It is evidently not an exception.

The first reflex is a ban by circular email. Its effect is familiar from other corners of hospital IT: experience shows the use hardly becomes rarer, only less visible — and thereby withdrawn from leadership.

Why shadow AI exists

The tools solve a real problem. Employed physicians spend, according to the MB-Monitor 2024 — the Marburger Bund physician survey — an average of around three hours per day on administration and documentation; a survey by the Deutsches Krankenhausinstitut (DKI), a German hospital research institute, from the same year likewise arrives at just under three hours [1]. Whoever still has four letters ahead of them at 7 p.m. and knows a tool that turns bullet points into fluent text will use it. That the hospital offers no vetted tool changes little; it shifts the use into private accounts, where nobody sees it.

For department leadership, this is first of all information about demand — and only after that a compliance case. Whoever reverses the order and starts with the sanction loses both: the insight into actual use and the chance to shape it.

What is actually at stake

Patient data in private accounts: a consumer account with an AI service is no place for patient data. Where the inputs are processed, whether they are reused for training, who gets to see them later — all of that lies outside the control of the hospital and its contracts. This rule needs no chain of legal citations; it only needs to apply without exception.

Unreviewed quality: error research on AI-generated clinical text draws a consistent picture. In a two-month real-world deployment with 7,545 AI-created notes, the systematically reviewed sample showed omissions in 18 percent and hallucinations in 11.5 percent of the notes [2]. The dominant risk is what is missing, and it is harder to see than what is invented. The Scientific Advisory Board of the Bundesärztekammer — the German Medical Association — states the consequence: AI results are not to be adopted uncritically; the responsibility remains medical [3].

The quiet legal duty: since 2 February 2025, Article 4 of the EU AI Act requires deployers to ensure the AI literacy of their staff; the Deutsche Krankenhausgesellschaft, the German Hospital Federation, additionally points to possible personal liability of decision-makers [4]. A department in which AI has in fact long been in use without anyone having been trained stands poorly on this duty — that the use ran "privately" is unlikely to be much consolation.

The playbook

1. Stocktaking without sanction

Two weeks, anonymous, three questions: which tools are used, for what, how often? Whoever ties the answers to consequences gets no honest ones. Experience suggests the result comes out broader than expected — and that is exactly why it is leadership information: it shows where the documentation burden is highest and which tasks staff perceive as automatable.

2. Few, clear rules

Three rules that anyone can recite in the corridor work better than a thirty-page policy: No patient data in private accounts, no exceptions. AI drafts are checked against the source before adoption. Whoever wants to use a new tool asks first — and gets a fast answer, otherwise the rule erodes.

3. A safe alternative in-house

A ban without an alternative lasts about as long as the shift stays quiet. The requirements for the alternative are the same as for any clinical tool: processing in the EU, contractually governed, drafts with source references, physician review as a fixed step. That is how aiomics builds its document generation: drafts whose statements link to the source in the record, with physician sign-off before any use — live in German rehabilitation clinics; extended review steps such as an omission detector that checks drafts against the record for what is missing are planned. Why verification is the more viable approach than ever more fluent generation is argued in the article "AI in healthcare: verification rather than generation" (in German).

4. Training that deserves the name

Article 4 of the AI Act requires it anyway; it becomes useful when it happens on your own material: real letters, real error classes, review heuristics for omissions, negations, numbers and attributions. Our CME-certified course on AI literacy (3 CME points, Ärztekammer Hessen — the Hessian state chamber of physicians) is one format for it.

5. Involve the Betriebsrat and data protection early

Whoever replaces tolerated private use with an in-house tool enters co-determination territory: if the employer provides the tool itself, §87 of the BetrVG — Germany's Works Constitution Act — is regularly triggered. The Hamburg labor court ruled in 2024 that merely permitting private ChatGPT accounts triggers no co-determination right of the Betriebsrat, the works council (decision of 16 January 2024 – 24 BVGa 1/24) [5] — the shadow configuration is thus, of all things, the co-determination-free one. What awaits you in an orderly introduction is covered in the article on introducing AI with the works council (in German).

What to measure after three months

1. The stocktaking can be repeated, and the answers shift from the private device to the in-house tool.
2. The rules can be quoted without looking them up — ask in the morning briefing.
3. A documented path for proposing new tools exists, and it has been used at least once.
4. The department's training rate can be evidenced — which is at the same time the proof for Article 4.

Shadow AI is, in the end, a leadership topic with a favorable starting position: the staff's motivation points in the same direction as the hospital's interest — what is missing is usually only the orderly path. If you would like a template for the stocktaking in your department, write to us. Ongoing analysis comes in our weekly briefing Visite (German; English edition Grand Rounds is in preparation).

Sources

1. Marburger Bund. MB-Monitor 2024 (Befragung von 9.649 angestellten Ärzt:innen, Herbst 2024): im Schnitt rund drei Stunden pro Tag für Verwaltung und Dokumentation. https://www.marburger-bund.de/bundesverband/themen/marburger-bund-umfragen/mb-monitor-2024-hohe-belastung-unzureichende ; Deutsches Ärzteblatt. DKI-Umfrage: Jeden Tag drei Stunden Bürokratie. 01.10.2024. https://aerztestellen.aerzteblatt.de/de/redaktion/dki-umfrage-jeden-tag-drei-stunden-buerokratie
2. Taylor SL, et al. Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics. 2026;14:e86474. doi:10.2196/86474
3. Bundesärztekammer, Stellungnahme des Wissenschaftlichen Beirats „Künstliche Intelligenz in der Medizin“, 14.01.2025. https://www.bundesaerztekammer.de/fileadmin/user_upload/wissenschaftlicher-beirat/Veroeffentlichungen/KI_in_der_Medizin_SN_neu.pdf
4. Verordnung (EU) 2024/1689 (KI-Verordnung), Art. 4; Deutsche Krankenhausgesellschaft, Positionspapier „Nutzung von Künstlicher Intelligenz (KI) im Krankenhaus“, 23.10.2025. https://www.dkgev.de/fileadmin/default/Mediapool/1_DKG/1.3_Politik/Positionen/2025-10-23_DKG-Positionspapier_KI_im_Krankenhaus.pdf
5. ArbG Hamburg, Beschluss vom 16.01.2024 – 24 BVGa 1/24. Besprechung: Legal Tribune Online. https://www.lto.de/recht/hintergruende/h/arbg-hamburg-24bvga-1-24-chatgpt-einsatz-arbeit-beteiligung-betriebsrat

Sources accessed in July 2026.