What AI Failure
Actually Costs

One billion people asked a health question online today.

A Stanford and Harvard study published January 2, 2026 found that AI medical models produce severely harmful clinical recommendations in up to 22.2% of cases. Even the best-performing systems made between 12 and 15 severe errors per 100 clinical encounters. The worst exceeded 40.

These two facts are rarely discussed together. They should be. Because the same week Google announced that billion-query figure at The Check Up 2026, new AI health tools were launching from Amazon, Microsoft, and Epic. At HIMSS 2026 two weeks earlier, clinical experts raised urgent concerns that AI agents being deployed across health systems had not been sufficiently validated with real patients.

The industry keeps shipping. The governance infrastructure has not kept pace. And the cost of that gap is no longer theoretical — it is accumulating in clinical outcomes, in financial exposure, and in the procurement rooms where health AI deals are dying for governance reasons that have nothing to do with the quality of the technology.

What Failure Actually Looks Like

The word "failure" in AI is doing work that needs to be unpacked. There is not one kind of health AI failure. There are at least four — each with a different cost structure, a different detection challenge, and a different governance requirement.

Errors of Commission

The most visible: the AI produces an incorrect recommendation, a wrong diagnosis, or a harmful treatment suggestion — with confidence. The NOHARM study found this in 22.2% of cases across the best available models. This is the failure mode that gets litigated.

Errors of Omission

More dangerous and harder to detect. The AI simply fails to flag what it should have flagged. Nobody notices the absence. This failure has no timestamp. It produces no audit trail. It accounts for 76.6% of the most harmful errors in the NOHARM study.

Calibration Failure

What happens when a system expresses high confidence in incorrect answers. A clinician who sees a high-confidence recommendation is less likely to apply independent judgment. Post-deployment monitoring research has consistently found that calibration degrades after deployment in ways that aggregate performance metrics completely mask.

Silent Drift

A model validated before deployment that has silently degraded as clinical practice, patient populations, or disease patterns evolved around it. A 2024 study of four top-performing mortality prediction models found that gradual performance decline was universal across all four post-deployment. The standard pre-deployment validation methods predicted none of it.

The Accountability Gap

HIMSS 2026 put the deployment velocity problem on full display. AI agents from Epic, Google, Microsoft, and Oracle were everywhere. The consistent message from the exhibition floor was speed and capability. The consistent message from clinical experts was: where is the validation?

Former FDA Commissioner Robert Califf, writing in a JAMA report on responsible AI in healthcare, was direct: no health system in the United States is currently capable of validating an AI algorithm once it is in use.¹ Not few health systems. Not most. No health system.

More than 1,200 AI-enabled medical tools have been cleared by the FDA. Most were evaluated on technical performance. The JAMA report found that clearance does not require demonstration of improved clinical outcomes. That gap between regulatory clearance and clinical benefit is where patients are currently living.

The Financial Costs Nobody Is Reporting

Direct Liability

An AI-generated clinical recommendation that contributes to patient harm creates liability exposure for the health system that deployed it — regardless of whether the AI vendor is also liable. Healthcare liability insurers are already noting that AI-related errors could lead to significant legal expenses and settlements. The insurance market has not yet fully priced this exposure. It will.

Operational Failure Costs

Completed-but-failed AI projects cost an average of $6.8 million while delivering only $1.9 million in value — a negative 72% ROI. Healthcare has a 78.9% AI project failure rate, second highest of any industry.²

Insurance Infrastructure Costs

Directors and Officers insurance carriers are now moving from voluntary AI governance questionnaires to mandatory governance endorsements — with hard mandates expected within 18 months. Hospitals without documented AI governance are already facing premium increases, specific exclusions, or loss of coverage entirely.³

The Consumer Dimension Nobody Is Governing

Google now processes one billion health queries daily. Amazon expanded its Health AI agent to Amazon.com in March 2026. Microsoft Copilot Health is live. These are not FDA-regulated clinical decision support tools — they are consumer information products with no post-deployment monitoring requirements and no clinical validation standard. And they are answering clinical-grade questions at a scale that dwarfs anything in the regulated medical AI space.

The question of who is responsible when a consumer receives incorrect health guidance, acts on it, and is harmed — is not yet resolved. It will be.

How This Plays Out in the Procurement Room

Hospital AI adoption requires a coalition: CFOs, COOs, IT leaders, legal and compliance teams, and clinical champions. Clinical enthusiasm is necessary. It is nowhere near sufficient.

The questions that consistently expose underprepared vendors are governance questions, not performance questions:

• What is your external validation data — collected from a population similar to ours?
• What does your performance look like across demographic subgroups?
• What is your drift monitoring plan, and how do you notify us when performance degrades?
• What is your change control process when you update the model?
• Do you have a model card or equivalent technical documentation?
• Who is liable when this tool contributes to an adverse outcome?
• What happens to our patient data if we terminate the contract?

A 2026 survey found 56% of medical group leaders have no formal AI governance policy and are not developing one.⁵ A separate survey found 72% of organizations have no software bill of materials for their AI models.⁶ The benchmark numbers get a vendor in the room. The governance documentation is what closes the deal — or kills it.

What Closes the Gap

Operational Proof requires validation in the actual deployment environment against real patient populations — not held-out test sets from the same academic medical center that generated the training data.

Runtime Monitoring requires continuous post-deployment surveillance designed to detect drift, calibration failure, and subgroup performance collapse before they produce adverse outcomes. The FDA's January 2025 draft guidance and the September 2025 Joint Commission and CHAI guidance both point the same direction: post-market surveillance is becoming a regulatory requirement, not a best practice.⁷

Accountability architecture means pre-assigning accountability for AI decisions, building infrastructure to audit AI recommendations, and defining what happens when performance falls below an acceptable threshold — before an adverse event forces the question.

The Question Worth Asking

One billion health questions a day. Severe errors in 22.2% of clinical AI recommendations. No health system capable of validating AI in use. More than 1,200 FDA-cleared tools evaluated primarily on technical performance rather than clinical outcomes.

The industry keeps shipping.

The organizations capturing the AI opportunity in 2026 are not the ones moving fastest. They are the ones that built governance infrastructure first and are now scaling on a defensible foundation. That is not a compliance story. It is a competitive one.