The dangerous mistakes AI is making when replying to patients on behalf of doctors

Researchers warn responses can include inaccurate medical advice and irrelevant information

Artificial intelligence is making dangerous mistakes when replying to patients on behalf of doctors – with medics often forced to rewrite the answers from scratch, a major study has found.

Researchers warn AI-generated responses can include inaccurate medical advice and irrelevant information a real doctor would immediately pick up. The findings raise fresh questions over the Government's rapid expansion of AI across the NHS, which was announced days ago by Health Secretary James Murray, promising the technology would free doctors from paperwork, speed up treatment and help cut waiting lists.

The move is part of NHS England's multi-billion-pound drive to accelerate the rollout of AI across the health service, including tools to help clinicians record consultations, triage patients through the NHS app, draft documents and cut administrative burdens. Officials insist AI is being used to support doctors, not replace them.

But the study shows there is a dangerous risk of mistakes in AI-generated drafts, summaries and patient messages that look polished but can miss crucial clinical warning signs. The new research also suggests the technology could cost doctors more time than it saves, with medics sometimes spending longer correcting AI-generated replies than it would have taken to write them from scratch.

Scientists discovered the systems frequently produced responses that were too long, included inaccurate or irrelevant medical information and failed to ask the follow-up questions that experienced clinicians should recognise as essential. One example involved a 32-year-old woman taking medication for acid reflux who contacted her doctor complaining of persistent nausea.

The AI-generated reply suggested she might need to adjust her diet while getting used to the medication. However, the doctor immediately deleted the advice and instead asked a crucial question the computer had failed to consider – whether there was any chance she could be pregnant.

Scientists said the example illustrated how AI can produce convincing medical language while overlooking vital clinical reasoning.
Researchers elsewhere have found examples where AI suggested inappropriate drugs for patients and others which failed to recognise situations needing urgent medical assessment.

The new research, carried out by Dartmouth College in the United States, analysed around 146,000 real conversations between more than 10,000 patients and their family doctor. It also tested responses generated by leading AI systems, including ChatGPT, Claude, Gemini, Llama, Aloe and Qwen. The researchers found AI-generated replies regularly failed to match what clinicians would actually write, often omitting important questions while adding unnecessary or inaccurate detail that doctors then had to remove.

Sarah Preum, assistant professor of computer science and one of the study's lead authors, said: "We find that AI can sound like a doctor but not think like one."

Prof Preum warned there was a danger of simply replacing one administrative burden with another, saying: "You don't want to integrate large language models into the workflow and just shift the bottleneck so that doctors are devoting their cognitive energy to playing AI janitor and fixing mistakes. But if we're not careful, that's a likely outcome."

Dr Tom Jefferson, a healthcare researcher at Oxford University’s Centre of Evidence-Based Medicine, said: “Medicine is not an exact science and you cannot trust a machine. I am extremely worried about the increasing use of AI in healthcare. AI makes mistakes and cannot distinguish between high-quality evidence and ‘noise’ (poor evidence). If you have to check it, you’re not saving time.”

Earlier research has also raised safety concerns. A study by Mass General Brigham and Harvard Medical School, published in The Lancet Digital Health, found 7.7 per cent of AI-generated patient replies could have caused severe harm.

And a 16-week 2025 Dutch study by the University Medical Centre Groningen found AI-generated draft replies failed to save clinicians significant time, largely because they still had to be carefully checked and edited. The findings have fuelled concerns about how quickly AI is being adopted across healthcare.

Critics warn AI can also invent facts, make factual mistakes and miss subtle clinical clues that experienced doctors would spot immediately. Co-author Dr Tim Burdick, a family doctor, said AI still required careful human oversight, adding: "I don't foresee a time when the portal can respond to a patient without a clinician editing it first."

He added lengthy AI-generated drafts could slow doctors down because they can contain information that is wrong, saying: "If you have to edit 75 per cent of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch," he said.

The researchers concluded every AI-generated response should continue to be checked by a clinician before it reaches a patient. Dr Burdick also stressed patients should not expect computers to replace doctors in the near future, stressing: "We're still nowhere near the point of having clinicians removed from the workflow."

The research team found AI could be improved by training it to mirror the communication style of individual doctors, boosting the quality of responses by around a third and reducing the amount of editing clinicians needed to do. Sir Jim Mackey, Chief Executive of NHS England, said: “The major overhaul of tech we’re making over the next few years will transform services. The new AI tool in the NHS App will help get patients to the best service for their needs the first time – whether that’s a GP appointment, trip to a pharmacy or advice on caring for themselves at home – so that clinicians can make sure those most in need of a GP appointment can get one sooner.

"We’re also seeing huge benefits from the introduction of AI notetaking tools, with clinicians finding they’re able to spend up to a quarter more of their time with patients, so we’re rolling out the tools as quickly as possible across the NHS. We’re prioritising the improvements that will make the biggest difference and supporting local leaders to adopt them to drive change in their services – helping to cut waiting lists and improve care for millions of patients so that the NHS is fit for the future.”