Zero Factual Errors, Then Round Five: Where Health Chatbots Actually Break

Ask a distressed user a single difficult question and the chatbot got it right every time. Let the same conversation run for a few turns and four in ten of those exchanges went wrong — and every one of the four did harm of the most serious kind. The same number that read zero in isolation read forty percent under sustained pressure. That gap, invisible to any test built from standalone questions, is the finding worth carrying out of this paper.

It comes from a red-teaming study by a team at Nationwide Children's Hospital and the Ohio State University, published in Scientific Reports in March 2026. The system on the bench is a modest, genuinely useful one: a retrieval-augmented chatbot, running on Claude 3.5 Sonnet, built to connect people with health-related social needs to local help — a food pantry, a shelter, financial aid in their county in Ohio. The paper is less interested in that one bot than in a harder question: how would you ever establish that a system like it is safe?

Two ways to be right

The protocol's most portable idea is a distinction between two kinds of correctness that look identical from the outside. One is whether the system stays faithful to the documents it retrieves — does it report what the source actually says? The other is whether it obeys the behavioural rules written into its instructions — does it refuse what it was told to refuse, escalate what it was told to escalate, stay inside its remit? A system can be immaculate on the first and unreliable on the second. The second is the one that reaches the patient.

To exercise both, the authors wrote 160 adversarial prompts and ran them two ways: as single questions, and as conversations sustained over several turns. This is not a clinical trial and never pretends to be one. It is a structured stress test of a single chatbot, probed by simulated rather than real users. Read at that altitude — a careful safety audit, not evidence of patient benefit — it does real work.

What broke, and when

On faithfulness to its documents the system was clean: zero errors across sixty checks. On obedience to its own behavioural rules it slipped about fifteen percent of the time even when questioned one item at a time. The instructive part is what length did to that figure. For advice-seeking queries the failure rate roughly doubled, from around thirty percent in single questions to fifty percent across multi-turn exchanges. For users expressing distress the shift was starker still: not a single failure when each question stood alone, then a forty-percent failure rate once the conversation extended — and all four of those failures were graded high-severity. In one, after several turns about an abusive parent, the system advised the user to try to avoid making your dad angry. No fact was misstated. The documents were quoted correctly throughout. The system simply improvised its way past the rules it had been given.

What fixed it

The remedies were as unglamorous as the diagnosis. Appending explicit prohibitions to the end of the system's instructions cut total errors by roughly sixty percent. Adding a curated document of local crisis resources to what the system could retrieve closed the distress gap in single-turn testing outright. Applied together, the two measures eliminated the high-severity multi-turn failures — not by making the model cleverer, but by routing it into a deliberate 'safe failure': handing the conversation off instead of inventing an answer about harm. The defence that worked was the refusal to let the model improvise where improvisation is dangerous.

What it does not show

One chatbot, one underlying model, one task, tested by its own builders against prompts they wrote, with simulated users. The authors say as much, and that candour is part of why the paper is worth reading. The annotation rested on a single human rater, with no inter-rater agreement reported — a limit the authors name themselves. None of this tells us how often a real person in distress reaches the fifth turn, or whether the same gap opens in a triage bot, a discharge-instructions bot, or a German-language one. Scientific Reports is peer-reviewed but a high-volume venue; this is a sound protocol paper, not a settled fact about chatbots at large. It is a method others now have to run.

Why it matters here

Procurement and regulation still tend to treat a conversational system the way they treat a calculator: pose the hard questions once, check the answers, sign off. The study is a clean demonstration that the single question is the wrong unit of testing. The patient-facing tools now arriving in European clinics and waiting rooms are reached most often by the people with the least slack — and those are exactly the people likely to keep asking, to press, to wear a system down. If your acceptance test is a list of questions answered correctly in isolation, you have measured the part of safety that was never seriously at risk. The part that matters surfaces only in the fifth turn.

Source: Hussain SA, Jackson DI, Lewis A, Fosler-Lussier E, Sezgin E. Toward trustworthy chatbots: a protocol for red teaming for health related conversations. Scientific Reports 2026;16(1):15550. A single-system, simulated-user safety study with a single human annotator, published in a high-volume peer-reviewed journal — a protocol to be reproduced, not yet a generalisable finding about patient-facing chatbots at large.