From human-in-the-loop to human-on-the-hook: The liability challenge of medical AI

A doctor sits across from a patient with an AI-generated recommendation on the screen. The system has analysed thousands of data points and produced a confident answer. The doctor has minutes to decide whether to accept it.

If the recommendation is correct, the technology has done what it was designed to do. If it is wrong and the patient is harmed, a harder question follows: who is responsible?

Increasingly, the answer may be the doctor. This is the central liability problem with medical AI. A clinician is kept “in the loop” to provide judgment and accountability. But the clinician may not know how the model was trained, what data it was tested on, where it performs poorly or why it produced a particular recommendation. The hospital, meanwhile, may have selected the product, negotiated the contract and decided how it would be used.

The doctor brought in to supervise the machine can end up carrying the risk created by it. The human in the loop can become the human on the hook.

When the machine sounds more certain than it is

The concern is not simply that AI makes mistakes. The more difficult problem is what happens when a machine’s answer causes a doctor to question a correct clinical judgment.

A 2024 study of 216 doctors found that some rejected their own correct diagnosis in favour of an incorrect AI suggestion in about 6 out of 100 cases. In radiology, incorrect AI recommendations caused specialists to reverse a correct diagnosis in about 7 out of 100 cases, according to trial data published on medRxiv in 2025.

The reason is partly the way newer systems communicate. Traditional diagnostic software often produced a score or limited output that a doctor could examine and challenge. AI systems can produce a complete explanation in fluent, authoritative language. An answer that sounds reasoned can be harder to question, particularly when a doctor is working under time pressure.

Training does not necessarily eliminate this problem. A randomised trial published in NEJM AI in 2025 found that physicians who completed structured AI-literacy programmes still showed a tendency to defer to flawed machine output when using the systems independently.

There is another problem. Large language models given clinical case vignettes containing a single incorrect detail produced false information in 50 to 82 out of 100 cases, according to a medRxiv trial published in August 2025.

Human oversight therefore depends on more than having a doctor present. The doctor needs enough information, time and authority to challenge the system.

Who carries the blame when AI gets it wrong

The law has yet to settle a basic question: when a patient is harmed after an AI-assisted decision, who should be held responsible?
The developer built the system. The hospital selected it and decided where it would be used. The doctor made the clinical decision. Each has exercised a different form of control.

Yet responsibility can still fall primarily on the physician. Medical Economics (2026) points out that doctors still have to make the clinical decisions, even as AI vendors are getting more scrutiny. Sommers Schwartz (2026) adds that there are also more lawsuits being filed, naming AI companies directly, particularly when companies have been unclear about the limitations of their systems.

A 2024 paper in Nature Humanities and Social Sciences Communications found that courts have no reliable method for apportioning liability among hospitals, vendors and physicians. A Stanford University brief cited in a 2025 arXiv paper similarly points to the difficulty of distinguishing a doctor’s error from a hidden defect in the AI itself.

This makes the hospital’s role difficult to ignore. Hospitals choose the technology, negotiate contracts, decide where it is deployed, determine what training clinicians receive and monitor how the system performs. They also decide how easily a doctor can override it.

This raises a fundamental question about whether a hospital can deploy an AI system and then treat every adverse outcome as an individual clinical error. The answer cannot rest entirely with the doctor who makes the final clinical decision. The hospital’s choice of technology, vendor, workflow, training and oversight has already shaped that decision long before the AI recommendation reaches the doctor.

The gap in India

India has an additional challenge because medical AI sits across several regulatory frameworks.

The drug regulator addresses whether medical software can be placed on the market and used. The National Medical Commission governs professional conduct. Data protection rules govern the handling of patient information. These areas overlap in practice, but the rules do not yet form a single framework for AI-related clinical liability.

As per Mavenrs (2026), in October 2025, India’s drug regulator issued draft rules classifying AI medical software according to risk. According to Doccure (2026), the standards of professional conduct for doctors when providing care thru digital means are still governed by the National Medical Commission rules. OC Academy (2026) states that the pace of AI adoption has surpassed medical education.

Data protection adds another layer. Healthcare organisations have compliance deadlines extending to May 2027 as per Chambers and Partners, 2026. Large hospital networks may have the resources to build the necessary systems and training programmes. Smaller clinics and healthcare startups face greater constraints.

The larger issue is the absence of a clear line of responsibility when the developer, hospital and doctor each control part of the decision.

What should change

AI companies should publish performance data that shows where their systems work well and where they fail. A single accuracy number tells a hospital very little if performance varies significantly between patient groups, clinical settings or types of cases.

Doctors also need a practical way to challenge AI recommendations. Systems should show limitations, flag uncertainty and make it easy to seek another assessment rather than presenting every output as a definitive answer.

Hospitals need to accept responsibility for deployment decisions. Vendor selection, contract terms, staff training, performance monitoring and the ability to override the system are governance decisions, not merely technical matters.

India also needs greater coordination between medical device regulation, professional standards and data protection requirements. A doctor should not be left to carry risks created by decisions made elsewhere in the AI supply chain.

Medical AI will become a routine part of clinical practice. The liability framework needs to catch up with the technology.

The real test of “human in the loop” is not whether a doctor is present when an AI system makes a recommendation. It is whether that doctor has the information, independence and authority to challenge it, and whether responsibility is shared by every party that puts the system into clinical use.