Is AI Healthcare's Newest Bureaucrat?
— Careful regulation can help prevent these tools from acting as gatekeepers to care
Featured Article | Originally published on MedPage Today by N. Adam Brown, MD, MBA
The U.S. healthcare system has officially entered the era of artificial intelligence (AI). Organizations, practitioners, and patients are no longer merely trying on a new fad, they are operationalizing the tool.
Consider these findings from KFF: about one-third of U.S. adults have used AI to find health information. More than 40% of health AI users say they have uploaded personal medical information into an AI tool.
In doctor's offices and hospitals, AI is used to write notes, triage messages, predict readmissions, summarize charts, suggest differential diagnoses, and even to make prescription decisions that, previously, a physician would have made. Ambient clinical intelligence has reduced the documentation burden and, in many cases, is improving the clinician experience.
While AI has the potential to improve patient outcomes, enhance healthcare access, reduce overall healthcare costs, and address clinician burnout, the industry also needs to ask some difficult questions about AI adoption. This applies not only to clinical settings, but also to insurance companies.
How Health Insurers Are Using AI
Health insurers are increasingly deploying algorithmic tools to evaluate claims, guide prior authorization decisions, and predict what care a patient "should" need. On paper, the rationale is straightforward: to improve efficiency, consistency, and cost control.
But clinicians are worried insurers are using AI to replace clinical review and with minimal human oversight. According to one survey from the American Medical Association (AMA), 61% of physicians said they fear that payers' use of unregulated AI has or will increase prior authorization denials. They are right to worry. In some reported cases, denial decisions were processed at a speed that makes meaningful physician review unlikely.
"Several cracks have emerged in the vision of a well-functioning, AI-driven insurance ecosystem," wrote Michelle Mello, JD, PhD, an empirical health law scholar at Stanford. "A major worry is that wrongful denials may be occurring as a result of a lack of meaningful human review of recommendations made by AI."
This problem should give all of us pause. Care decisions are not a probabilistic exercise in the same way many AI systems are designed to operate. AI systems are trained on large populations, optimized for pattern recognition, and deployed to generate recommendations -- or decisions -- based on statistical likelihood. When used appropriately, this assessment can be powerful. When used improperly, it can be dangerous. Patients are more than the sum of a few numbers; they are individuals who do not always map cleanly to a model.
And yet, that is exactly how some insurers are treating them.
What Happens When AI Is Used as a Gatekeeper, Not A Tool
The core issue is not that AI is being deployed in healthcare settings. It is how it is being used. There is a fundamental difference between viewing AI as a tool to enhance human skill and knowledge and using AI as a gatekeeper for care.
A tool supports human judgment. A gatekeeper replaces it.
As the evidence in the previous section demonstrates, we are seeing a quiet but meaningful shift toward the latter formulation. Fortunately, regulators are starting to take notice.
CMS has stated that while algorithms may be used to assist in coverage determinations, they cannot override individual patient circumstances or substitute for clinical judgment. The AMA has called for greater oversight of insurer AI use, emphasizing transparency, bias mitigation, and the need for human review in decisions that affect patient care. Patients are beginning to push back too, waging lawsuits that allege that AI-driven decisions have inappropriately denied care.
As a result of these moves, questions about insurer and practitioner use of AI has now turned toward liability. Who is responsible when an AI system contributes to a bad outcome? If a clinician follows an AI recommendation that leads to a poor health outcome, is that an error of clinical judgment -- or an algorithmic one? If an insurer uses AI to deny care and that denial leads to harm, where does accountability sit? With the insurance company? The AI developer? The data provider?
Limited Legal Frameworks for AI in Healthcare
AI introduces a new layer between information and action, one that is often opaque, difficult to interrogate, and constantly evolving. These features are deliberately built into AI systems.
Accounting for these realities is one reason the legal framework for AI has not caught up to the technology. As discussed in recent academic work, including analysis from Harvard's Petrie-Flom Center, liability in the age of AI is deeply uncertain because traditional models of malpractice assume a human decision-maker. AI eliminates that feature.
The healthcare system needs a clear set of boundaries. Regulators need to answer questions like: should AI be used autonomously? Where must a human remain as the decision maker? What level of transparency is necessary for clinicians and patients to trust these systems? And critically, who is accountable when things go wrong?
If we do not define the role of AI in healthcare, it will define itself -- driven by incentives that will not always align with patient care.
In the clinical environment, the path forward is relatively clear. AI should augment, not replace, clinician judgment. Outputs should be reviewable, explainable, and contestable, and the clinician should remain the final decision-maker.
In the payer environment, the standards need to be just as strong, if not stronger. Coverage decisions cannot be reduced to algorithmic outputs without meaningful human oversight. Models must be transparent in how they are used, validated against clinical standards, and monitored for bias. And there must be clear accountability when it comes to impact. Otherwise, we risk building a system where decisions that shape access to care are made by tools that no one fully understands and no one fully owns.
AI is not the future of healthcare. It is our present. The question now is whether we build a system where it serves patients — or one where patients are forced to serve the system.
It's still early enough to decide. But not for long.
Adam Brown, MD, MBA is a physician, healthcare executive, and business professor. He is the founder and managing partner of ABIG Health, a Washington, D.C.-based healthcare strategy firm.
