Software Selection: Five Demands That Give Chief Physicians a Say

The invitation arrives as a calendar entry: "Market screening, AI documentation — clinical input requested." Forty-five minutes, between the morning meeting and the OR schedule, attached a vendor deck of forty slides. The temptation is to pass the appointment on to the senior physician who is "good with IT." It would also be the mistake.

Because procurement gets decided — with or without a medical voice. If the medical directorate is not at the table, price, reference list and slide design decide. Yet clinical input requires no tendering expertise. Five demands suffice. They are vendor-neutral, can be formulated in a single meeting, and they separate vendors faster than any feature list.

First: every statement shows its source

Demand that every generated sentence — in the draft letter, in the coding suggestion, in the summary — points to the place in the source material it comes from. One click, in the document itself.

The reasoning is arithmetic: checking without source pointers means researching from scratch. If the physician has to search the record herself for every statement, the review effort eats the saving — and in practice, at some point, things get accepted unchecked. A sentence without a source location is, in case of doubt, an assertion by the system — and should look that way in the document: visibly different from evidenced text.

Second: a trial run on your own records

Error rates from studies and vendor material are barely comparable with one another. Some count only factual inventions, others also omissions and inconsistencies; depending on the definition, the reported rates of modern systems sit at roughly one to three percent — with the authors' caveat that in healthcare even small percentages can have large consequences [1]. The only number that counts for your hospital comes into being in your hospital.

So demand a time-limited trial run with your own case mix: your own document types, your own scan quality, your own terminology — a 14-day test protocol (in German) can be fixed in advance. Define before day one what gets counted: omissions, inventions, mix-ups of persons or laterality, each separately. Define, too, when to stop — an abort criterion nobody wants to name in advance is one nobody will apply afterwards. And: a vendor who refuses the test on your own records, or insists on curated demo cases, has thereby answered the question.

Third: sign-off is technically enforced

People adopt suggestions from systems they trust — wrong ones included. In a prescribing study, physicians switched from a correct to a wrong answer in 5.2 percent of cases after wrong advice from the system [2]. In a radiology experiment, the accuracy of even very experienced readers fell from 82 to 45.5 percent under wrong AI suggestions — obtained on vignettes under experimental conditions, but a clear signal that experience alone does not protect [3].

That is why the standing instruction "please check carefully" is not enough. Sign-off must be enforced in the software: no document leaves the system without a named physician's sign-off, no batch confirmation across whole stacks, changes remain traceable. Ask literally: "Is there an accept-all button?" The right answer is no.

Fourth: the data path can be named

Where does processing run — region, infrastructure, subprocessors? Is your data used for training, and is the exclusion in the contract or only in the brochure? What deletion periods apply to audio and intermediate products, and can deletion be evidenced?

That sounds like a task for the legal department, but it concerns everyday clinical work: these are your patients' conversations and your department's records. A vendor who cannot describe the data path in five sentences either does not know it or does not want to describe it. The contractual side of these questions is a discipline of its own — the typical weak points in data processing agreements (in German) we have written up separately.

Fifth: the exit is described

What happens at the end of the contract? Demand a concrete answer: export of all content in open, reusable formats, transition periods for parallel operation, a rule for the facility-specific templates and configurations that have grown over years. The configurations especially get underestimated: the letter templates, checklists and form mappings a department builds up in two years of operation are the system's real value — and in the worst contract they belong to nobody. The better a tool is integrated into daily work, the more expensive an unplanned departure becomes — and the more important it is that the planned one stays possible. A vendor who has thought through its own replacement thinks in a hospital's time spans.

What the list achieves

None of the five demands requires technical training, and no serious vendor can object to any of them. It gets interesting in the concrete: "Show me sentence and source." "Let us test on our own records." "Show me how sign-off is enforced." "Describe the data path." "Describe the exit." Whoever carries these five sentences into the procurement round does not have to run the tender — the criteria do it.

There is also nobody else in that round who could take over the role: IT checks interfaces, purchasing checks prices and contract terms, the legal department checks liability and data protection. The question of whether a tool holds up in everyday clinical work — with real cases, under time pressure, with tired reviewers — is answered by none of these departments. It remains a medical question, whether it gets asked or stays unasked.

And if the forty-five minutes from the calendar entry are not enough: they are enough for the five sentences.

If you want to follow how selection criteria and the evidence base develop: in our weekly briefing Visite (German; English edition Grand Rounds is in preparation), we sort both — briefly and with sources.

Sources

1. Topaz M, Peltonen LM, Zhang Z. Beyond human ears: navigating the uncharted risks of AI scribes in clinical practice. npj Digital Medicine. 2025;8(1):569. doi:10.1038/s41746-025-01895-6.
2. Goddard K, Roudsari A, Wyatt JC. Automation bias: empirical results assessing influencing factors. International Journal of Medical Informatics. 2014;83(5):368–375. doi:10.1016/j.ijmedinf.2014.01.001.
3. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology. 2023;307(4):e222176. doi:10.1148/radiol.222176.