← Weekly AI Healthcare News

Healthcare AI Weekly Deep Dive

October 09 - October 16, 2026
Three storylines define this week. First, the prior authorization arms race just got a new combatant: Otter Health raised $8M to sidestep the whole PA fight by reshaping care decisions upstream, while Centene's gold card program signals that even payers are tired of their own systems. Second, agentic AI governance is getting serious attention from the AMA and AHA simultaneously, with real frameworks landing as health systems scramble to figure out who's liable when an AI agent makes a clinical call. Third, Anthropic had a monster week everywhere except the one place your clients actually operate: the clinical workflow. Workday, HCLTech, and Hexaware all signed Anthropic deals, which means Claude is coming to the enterprise HR and back-office layer of every major health system whether your clients want it or not. The Suki $90M Series E and Confido Health's $12.5M extension confirm the market still has appetite for clinical AI agents, but the real question is whether any of these companies can survive the Epic ecosystem. UnitedHealth's Q3 call dropped a quiet bombshell: AI recommendations are now live inside Medicare Advantage plans. That sentence should be in every client deck this week.
In This Issue
Top Stories
Act Now

UnitedHealth Puts AI Inside Medicare Advantage Decisions

UnitedHealth just embedded AI recommendations inside the plan that covers 8 million seniors. This is not a pilot. Every system negotiating MA contracts right now is negotiating against an AI-augmented payer.
UnitedHealth reported $4.1 billion in Q3 profit and buried the real story in the earnings call: AI recommendations are now live inside Medicare Advantage plans. Read that again. This is not a roadmap item. It's not a press release about a pilot. It's a deployed system making recommendations inside plans that cover millions of seniors. Here's the thing. When a payer with UNH's scale puts AI inside the MA decision layer, it changes the negotiating dynamic for every health system at the table. The AI model has one optimization target: UNH's financial performance. Not patient outcomes. Not your HEDIS scores. Not your quality bonus thresholds. UNH's. The second-order consequence nobody is talking about: AI recommendations make the appearance of clinical judgment cheap. Which means fake clinical judgment also gets cheap. UNH can now generate a recommendation backed by a model trained on 8 million MA members and present it to a plan participant or a provider as something that looks like individualized clinical guidance. Your clients have no visibility into that model's architecture, training data, or known failure modes. Call this what it is: a black box inserted between your care management team and your patients. The consulting play here is immediate. Clients need a rapid assessment of every MA contract that touches UNH. They need to understand what data UNH is using to generate these recommendations, whether those recommendations will surface in prior auth workflows or care navigation tools, and what the appeal pathway looks like when an AI recommendation drives a denial. This is also a risk adjustment story. If UNH's AI is identifying coding gaps and surfacing them as recommendations, that has direct revenue implications for any system running a risk adjustment program. The $4.1B profit number tells you UNH has the margin to fund aggressive AI deployment. Your clients do not have the margin to ignore it.
Risk angle: AI recommendations inside MA plans almost certainly skew toward lower-cost interventions. If the model's training data reflects UNH's historical denial patterns, your clients are now fighting an algorithm with a built-in financial bias and no published methodology for appeal.
Act Now

AMA's Agentic AI Framework Lands. Nobody Is Ready.

The AMA just handed every health system a governance liability checklist disguised as a framework. If your client's agentic AI deployment doesn't map to these principles and something goes wrong, plaintiff attorneys now have a published standard to cite.
The AMA dropped a governance framework for agentic AI in healthcare this week, and it landed quietly. That's a problem. Because the AMA saying agentic AI must earn clinical authority through evidence is not a philosophical statement. It's a liability marker. Here's what actually happened this week: the AMA published a framework, and simultaneously, AHA and Anthropic were actively shaping state-level sandbox regulations. Both moves are going in the same direction. The professional and hospital associations are building the governance architecture because federal regulation isn't coming fast enough. Which means the standards that matter are the ones your clients will be held to in court, in state audits, and in Joint Commission reviews. Not the FDA. Not CMS. The AMA framework reportedly requires that agentic AI demonstrate evidence of clinical benefit before taking clinical action, that humans remain in meaningful oversight roles, and that liability frameworks are clear before deployment. Every one of those requirements is a consulting engagement. When the lab result fires automatically and routes a patient to a specialist, who approved that? When the care management agent sends a discharge follow-up message, what's the documentation trail? When the billing agent codes a visit, who reviewed it? These aren't hypotheticals. Confido Health just raised $12.5M to deploy agentic AI across PE-backed medical groups. Carenet just acquired Invene to build agentic AI for Fortune 100 payers. The deployment is happening. The governance is not keeping pace. The AMA framework gives your clients a starting point. The consulting play is taking that framework and translating it into actual workflow documentation, role definition, and vendor contract requirements. Agentic AI governance is the 2026 equivalent of HIPAA compliance in 2002. The early movers will build the playbook. The late movers will buy it from the plaintiffs' bar.
Risk angle: Frameworks without enforcement are just paper. The AMA has no regulatory authority. But in litigation, a published professional standard from medicine's largest trade organization becomes the de facto duty-of-care baseline. Any system deploying agentic AI without documented compliance with this framework is taking on legal exposure they may not understand.
Watch This

Suki's $90M Bet: AI Agents, Not Just Scribes

Suki is using $90M to pivot from ambient scribe to full AI agent platform, which means the ambient documentation market is about to collapse into a broader clinical workflow competition. Your clients who just signed three-year ambient scribe contracts need to understand what they bought.
Suki just raised $90M in a Series E round and the stated use is AI agents, not ambient scribes. That's the tell. The ambient documentation market is mature enough that $90M in growth capital can't be justified on scribe expansion alone. Suki has 30 health technology partners and is betting that its existing clinical workflow presence becomes the on-ramp for autonomous agents. This is the right strategic move and probably two years too early. Here's the thing. The ambient scribe solved a real problem: physicians hated typing. Agents solve a different problem: health systems are drowning in tasks that don't require a clinician but currently land on one anyway. Prior auth requests. Referral coordination. Care gap outreach. Medication refills. The agent play is targeting the clinical administrative burden that scribes never touched. But the path from scribe to agent is not obvious. Suki's competitive moat is clinical context. The scribe hears the visit. The agent that acts on that visit needs to know the payer rules, the patient's prior auth history, the preferred pharmacy, and the referring physician's fax number. That's not a transcription problem. It's an integration problem. Suki's 30 partner relationships are the real asset here, not the language model. The consulting question for your clients: where does Suki's agent play sit relative to Epic's own agent ambitions? Epic controls the chart. Epic controls the workflow. Epic is building its own agent layer. Any third-party agent platform is either going to partner deeply with Epic or fight for the margin Epic doesn't want to serve. Suki's existing Epic integration is its best defensive moat. Whether $90M is enough to build out before Epic closes the door is the real question.
Risk angle: Ambient scribe makes note-writing cheap, which means the bottleneck shifts to the clinician holding patient context across 14 workflows. Suki's agent play assumes that solving documentation unlocks agent deployment. That's not obvious. The same integration and trust problems that slow scribe adoption are ten times harder for autonomous agents.
Watch This

Epic's Drug Pricing Move Is Bigger Than It Looks

Epic just embedded price transparency into the prescribing workflow, which is the one moment where drug pricing decisions actually get made. This is not a patient-facing app. It's a prescriber-facing tool that reshapes formulary decisions at the point of care.
Epic partnered with Amazon Pharmacy, Eli Lilly, and Mark Cuban's Cost Plus Drugs to put real-time drug pricing inside the physician prescribing workflow. Every CIO is going to get a deck this week calling this a patient benefit. It is. But that's the least interesting part of what just happened. Here's what Epic actually did: it inserted itself into the pharmacy supply chain at the exact moment the prescribing decision gets made. Not after. Not at the pharmacy counter. At the point where the physician picks the drug and the dose. That's the moment that controls everything downstream. The parties in this deal are not random. Amazon Pharmacy wants to capture the prescription fill. Eli Lilly wants its GLP-1s to look price-competitive against biosimilars. Cost Plus wants its generic pricing to make brand drugs look expensive. Epic wants to be the operating system for the entire care delivery experience, including the pharmacy transaction. The PBMs are not in this deal. That is not an accident. This is a direct shot at the PBM model. If physicians start defaulting to Cost Plus or Amazon Pharmacy because the price comparison shows up natively in Epic, PBM formulary leverage erodes. For health systems with 340B programs, this creates a real problem. The 340B model depends on capturing prescription fills through system-affiliated pharmacies. If Epic's tool routes patients to Amazon Pharmacy at point of prescribing, that capture rate drops. Health systems need to audit this before it goes live in their Epic environment and understand whether they can configure the tool to prioritize their own pharmacy channels. This also has implications for any system with a PBM contract that includes preferred formulary arrangements. Those arrangements assume physician behavior that Epic may now be redirecting.
Risk angle: Amazon Pharmacy wins if Epic physicians default to its channel. Eli Lilly wins if its drugs surface favorably in the price comparison. Mark Cuban's Cost Plus wins if transparency erodes brand loyalty. The only party with no obvious win condition is the PBM sitting between all of them.
Watch This

Prior Auth Arms Race Gets a New Weapon

Otter Health is betting that influencing the care decision before it happens is smarter than fighting the PA denial after. If it works, it breaks the current AI arms race model and creates a new one around real-time clinical quality.
Otter Health raised $8M to build what it's calling a real-time clinical quality platform: instead of automating the PA request after a care decision is made, Otter tries to shape the care decision so it never triggers a PA requirement in the first place. The framing is clever. The execution is hard. Here's the genuine upside: the PA arms race is a treadmill. AI makes PA submission cheap, which means payers get buried in submissions and harden their denial criteria. AI makes appeal automation cheap, which means payers auto-deny more. The cycle accelerates and no value gets created. Otter is trying to step off the treadmill. The genuine downside: this model requires deep clinical workflow integration, real-time data access, and payer acceptance of a fundamentally different model for utilization management. None of those things are easy. The payer acceptance piece is the biggest unknown. Centene just launched a gold card program, which is the payer acknowledging that the PA system is broken for high-trust providers. That's directionally consistent with what Otter is selling. But gold cards are based on provider track record, not real-time clinical quality signals. Otter's model asks payers to trust its platform's real-time assessment instead. That's a much harder sale than a gold card program. For health systems evaluating PA strategy: the portfolio answer is probably both. Automation tools for the denials you can't avoid, and upstream clinical quality tools for the care patterns where you have enough volume to make a payer deal. Otter is worth a watch, not a contract, until payer acceptance is demonstrated with a named health plan.
Risk angle: This is a trap for systems that want to skip the arms race entirely. Influencing care decisions upstream sounds better than fighting denials downstream, but it requires clinical workflow integration that is just as hard as building the appeal automation. And if the payer doesn't accept Otter's clinical quality data as a substitute for PA, you've built a great tool that saves zero PA headaches.
VBC Watch
Watch This

Diverge Health Plus Medical Home Network: Safety Net VBC Gets Scale

The combined Diverge Health and Medical Home Network entity enters 2027 as one of the largest VBC platforms dedicated to safety-net populations. For consultants working in Medicaid and FQHC strategy, this changes the competitive landscape for VBC technology and care management support.
Diverge Health and Medical Home Network announced a merger that creates one of the largest value-based care platforms exclusively focused on safety-net and underserved communities. The combined organization enters 2027 with meaningful scale in a market that has historically been underserved by VBC technology vendors who built for well-resourced health systems first. The strategic logic is sound. Safety-net populations carry the highest disease burden, the most complex social determinants of health, and the greatest opportunity for VBC savings if care is coordinated effectively. They also have the most fragile funding base and the least infrastructure for the data exchange that VBC requires. The technology angle here is the one your clients need to watch. Medical Home Network has built real analytics capabilities on top of claims and clinical data for Medicaid populations. Diverge brings care management execution. The combined platform could become the reference architecture for FQHC and safety-net VBC programs nationally. For consulting clients in Medicaid managed care, this merger creates a new category of competitor and partner. A combined entity with this population data depth and care management reach can go to payers with risk-sharing arrangements that individual FQHCs cannot. It also creates a template for other safety-net consolidation plays. The VBC technology funding environment is tight, as Healthcare IT News reported this week that ambitions are outpacing confidence in funding. A merged, scaled entity with demonstrated outcomes data is better positioned to attract the remaining capital than two smaller organizations competing for the same pool.
Risk angle: Safety-net VBC is politically and financially fragile. Both organizations depend on Medicaid reimbursement structures that are under active pressure from federal budget discussions. A merged entity with more overhead than either had alone is more exposed if Medicaid rates move.
Act Now

UNH's AI Risk Adjustment Play Is Operational, Not Theoretical

Risk adjustment AI is moving from point solutions to operational systems embedded in payer workflows. Health systems that aren't running equivalent AI on their own RAF data are losing ground in a zero-sum coding accuracy competition.
The risk adjustment AI story is moving fast and most health system clients are still in evaluation mode while payers are in production mode. UNH's Q3 earnings this week confirmed AI is live inside Medicare Advantage plan recommendations. Risk adjustment is almost certainly part of that deployment. Here's the structural problem. Risk adjustment is a zero-sum game. RAF scores determine plan payments from CMS. Payers have enormous financial incentive to identify coding gaps that increase RAF scores, which increases their capitated payments. Providers have enormous financial incentive to document accurately and capture revenue they've earned. When payers deploy AI to identify coding gaps and surface them as care recommendations, and providers are still doing manual chart reviews, the information asymmetry is massive. The consulting play is straightforward. Clients need a risk adjustment AI vendor assessment alongside their current process audit. Point solutions for HCC coding are mature and there are real vendors doing this well. The shift the Becker's article is describing is from those point solutions to integrated operational systems that run continuously, not as periodic chart pull exercises. For ACO and VBC clients specifically: risk adjustment accuracy is not a back-office function anymore. It's a care quality function. When a patient has an undocumented chronic condition, the care plan is wrong and the RAF score is wrong simultaneously. The AI tools that fix both problems at once are the ones worth buying. Separate tools for coding and care gaps are the old model.
Risk angle: AI risk adjustment makes coding gap identification cheap, which means fake gap closure also gets cheap. Both sides of the payer-provider relationship now have AI generating coding recommendations, and CMS audits can't keep pace. The RAF inflation risk is systemic.
M&A & Partnerships
Watch This

Carenet Buys Invene: Agentic AI for Fortune 100 Payers

Carenet just bought an AI and data engineering consulting firm specifically to build agentic AI for the payer side. This is the twelfth acquisition in their portfolio. They're building the payer-side AI stack and they're moving fast.
Carenet Health acquired Invene, a healthcare-only AI and data engineering consulting firm, marking its twelfth acquisition and its second in the same week after buying conversational AI platform Orbita. The stated target is Fortune 100 payers. Let's call this what it is: Carenet is building a vertically integrated AI services firm for the payer market. Clinical execution, member engagement, conversational AI, and now data engineering and agentic AI strategy. The combination is formidable. Carenet already has payer relationships. Invene brings the technical depth to build custom agentic AI on top of those relationships. The consulting implication is direct. If your practice serves payer clients on AI strategy or digital transformation, Carenet is now a competitor with a lower cost structure, existing client relationships, and a faster deployment track record than any of the Big 4. The Invene acquisition also signals something about where the healthcare AI consulting market is going. Pure-play consulting without implementation capability is getting squeezed. Carenet is betting that owning the full stack, from strategy to data engineering to clinical execution, is the defensible position. For health system clients: understand that the payer you're negotiating with may be getting agentic AI strategy advice from a firm that also runs member engagement programs designed to optimize the payer's cost outcomes. That's a complicated relationship to navigate.
Risk angle: Carenet is simultaneously a clinical execution firm and now an AI consulting firm targeting the same payers your clients negotiate with. That creates a conflict of interest dynamic worth watching. The firm advising your payer client on agentic AI strategy also runs the call center agents that talk to their members.
Watch This

Anthropic Signed Three Enterprise Deals This Week. Zero Were Clinical.

Anthropic landed partnerships with Workday, HCLTech, and Hexaware this week, all enterprise back-office plays. Claude is coming to health system HR, finance, and operations through the Workday layer before it gets anywhere near clinical workflows. Your clients may already be buying it.
Anthropic had a banner week for enterprise deals: Workday, HCLTech, and Hexaware all announced Claude partnerships. None of them are clinical AI plays. All of them are paths into health system operations through the enterprise software layer. The Workday deal is the most significant for your clients. Workday runs workforce management, financial planning, and HCM for hundreds of health systems. When Workday embeds Claude in its agentic AI layer, which it confirmed this week, health system employees are using Anthropic's model for scheduling, budget planning, and HR decisions. The clinical governance team didn't approve this. The CMIO didn't review it. It's happening through a software update to an existing contract. HCLTech's Anthropic deal is a different vector. HCLTech is a major implementation and managed services partner for health systems globally. They're building a dedicated Anthropic business unit. That means the implementation firm your clients use for EHR optimization and IT infrastructure is now actively selling Claude-powered services into your clients' organizations. The Hexaware multi-year partnership follows the same pattern. This is consent laundering. Health systems are getting Anthropic capabilities deployed in their organizations through vendor relationships that don't trigger clinical AI review processes. The answer is not to block it. The answer is to build an AI governance process that covers enterprise software procurement, not just clinical tool evaluation. Every new Workday capability, every new managed services contract, every new ERP update needs to go through the same AI inventory process as a clinical decision support tool.
Risk angle: Workday runs HR and finance for most large health systems. If Claude is embedded in Workday's agentic AI layer, health system employees are using Anthropic's model for workforce management, financial planning, and operational decisions. Nobody asked the CMO about this.
Watch This

Agilent Buys Proscia for $270M: Digital Pathology Gets Serious

Agilent is buying Proscia's digital pathology AI platform for up to $270M, which consolidates the digital pathology software market under a major lab instrumentation vendor. Health systems building digital pathology programs need to understand how this changes the vendor landscape.
Agilent Technologies is acquiring Proscia for up to $270M in an all-cash deal, $240M at close plus $30M contingent on operational milestones. Proscia built Concentriq, a digital pathology AI platform used by health systems and research institutions to manage, analyze, and collaborate on pathology slides. Agilent makes the instruments that generate those slides. The strategic logic is vertical integration: own the scanner, own the software, own the AI that reads the output. For health systems: digital pathology is one of the most real, deployed clinical AI applications in medicine right now. Computer vision for tissue analysis has genuine clinical validation, unlike many other healthcare AI claims. Proscia's platform has been the independent software option for systems that didn't want to be locked into a scanner vendor's ecosystem. That independence is now gone. The acquisition pattern here mirrors what happened in radiology AI: hardware vendors and large platform players have been consolidating independent AI software companies. The independent PACS-adjacent AI companies got acquired or squeezed. The same dynamic is now playing out in pathology. For health system pathology programs evaluating digital pathology infrastructure: the Agilent acquisition changes the competitive dynamics. Systems running Proscia should assess their contractual protections and understand Agilent's product roadmap before the next contract cycle. Systems evaluating digital pathology platforms should add 'independence from scanner vendor' as an explicit evaluation criterion.
Risk angle: When a hardware vendor acquires the software platform, software development priorities shift toward hardware compatibility and margin protection. Health systems that chose Proscia for its independent software roadmap may find that roadmap now bends toward Agilent's instrument sales strategy.
Consulting Intelligence
Watch This

Carenet Is Building the Payer AI Consulting Stack Your Clients Will Hire

Carenet is executing a vertical integration strategy in healthcare AI services that positions it to compete directly with traditional consulting firms on payer AI engagements. Two acquisitions in one week signals urgency.
Two acquisitions in one week is not a coincidence. Carenet acquired Orbita, a conversational clinical AI platform, and Invene, a healthcare AI and data engineering consulting firm, in the same seven-day window. This is a deliberate build-out of a vertically integrated AI services firm targeting Fortune 100 payers. The traditional consulting model for payer AI work is: assess the problem, define the strategy, hand off to an implementation partner, collect the fees. Carenet's model is: assess, build, deploy, operate, and collect recurring revenue from the payer relationship indefinitely. That's a fundamentally different economics model and it's winning with buyers who are tired of the assess-and-hand-off cycle. For traditional consulting firms with payer practices: the threat here is not that Carenet has better ideas. It's that Carenet has implementation credibility that pure advisory firms cannot match. When a Medicaid plan is choosing between a consulting firm that will tell them how to build agentic AI member engagement and an operational firm that already runs member engagement and is adding agentic AI to existing workflows, the friction advantage is enormous. The counter-positioning for traditional firms is governance, independence, and cross-industry perspective. Carenet can't tell a payer what a hospital system thinks about agentic AI member engagement. A consulting firm with both payer and provider clients can. That's the defensible ground. Stake it explicitly.
Risk angle: Carenet's model blends advisory and operational execution in ways that traditional consulting ethics frameworks are not built to handle. A firm that advises on AI strategy and operates the member engagement programs that implement that strategy has a structural conflict of interest.
Watch This

AI Governance Is the New HIPAA. Who's Building the Playbook?

State AI sandbox regulations are being written right now, and the industry groups shaping them will define what compliance looks like for the next decade. The consulting firms that build governance frameworks during the rulemaking window will own the compliance work that follows.
STAT News reported this week that advocacy groups are scrambling to shape the regulations governing state AI sandboxes. Simultaneously, AHA and Anthropic are influencing state oversight of autonomous clinical AI. The AMA dropped a governance framework. Thirteen states have healthcare AI laws already on the books according to Becker's. Readily raised $11M specifically to build an AI regulatory compliance platform for healthcare. All of this is happening at once. That is not a coincidence. This is the pre-regulatory window, and everyone who is going to shape what compliance means is doing it right now. The consulting play here is one your firm can't afford to miss. When HIPAA was being drafted and operationalized in the late 1990s, the consulting firms that built privacy frameworks during the rulemaking window owned the compliance market for a decade. Every hospital needed a HIPAA gap assessment, a remediation roadmap, and ongoing compliance support. AI governance is going to follow the same arc. The difference is that AI governance is more complex, more technically demanding, and more clinically consequential than HIPAA. The firms that build credible AI governance frameworks now, before the regulations are finalized, will define what 'compliant' means for their clients. So here's the challenge. If you run a healthcare consulting practice: one, get someone on the state sandbox regulatory proceedings in your key markets. Two, build a healthcare AI governance framework that maps to AMA principles, state sandbox requirements, and emerging federal guidance simultaneously. Three, position that framework as a product, not a custom engagement. Four, don't wait for a client to ask. The clients who need this most don't know they need it yet.
Risk angle: Sandboxes that look permissive in 2026 may generate liability exposure in 2028 when the guardrails tighten and health systems have already deployed at scale under looser rules. Building for the sandbox rather than for the eventual standard is a classic pilot theater move.
Did You Know?

Workday and Anthropic's Deal Shows How Claude Enters the Enterprise

Workday and Anthropic announced a partnership this week to embed Claude's reasoning capabilities inside Workday's agentic AI platform. Workday runs human resources, workforce management, financial planning, and supply chain for hundreds of large health systems. The partnership means that the next time a health system's Workday environment gets an AI capability update, it may be powered by Anthropic's Claude model. Here's what makes this educationally important for healthcare AI strategy: this is a fundamentally different procurement pathway than clinical AI adoption. When a hospital buys an ambient scribe or a sepsis prediction tool, it goes through clinical validation, CMIO review, IT security assessment, and often a formal vendor contracting process with clinical performance guarantees. When Workday adds an AI capability to its existing platform, it typically goes through software procurement review, not clinical AI governance. The two pipelines don't talk to each other. This creates a gap that is showing up everywhere in healthcare AI deployment: AI tools that affect clinical workflows, scheduling, and patient-facing operations are entering health systems through enterprise software relationships that don't trigger clinical review. Scheduling AI that determines nurse staffing ratios affects patient safety. Financial planning AI that models service line profitability affects care access. HR AI that influences hiring and retention decisions affects workforce composition. None of these are clinical AI tools by traditional definition, but all of them affect clinical outcomes. The Workday-Anthropic deal is a case study in why healthcare AI governance frameworks need to cover enterprise software procurement, not just clinical tool selection. The firms building those frameworks now are the ones that will be called when something goes wrong.

Mount Sinai Found a Simple Fix for AI Safety. It Worked 19 of 20 Times.

Mount Sinai researchers tested whether simply adding safety-focused system prompts to AI model queries could reduce harmful clinical recommendations. The answer was yes, in 19 of 20 models tested. This finding matters for health system AI strategy for a reason that goes beyond the headline result. The finding suggests that a significant portion of AI safety risk in clinical settings is not model architecture risk. It's prompt engineering risk. Models that produce harmful outputs without safety prompts produce substantially fewer harmful outputs with them. That's a governance and deployment finding, not a fundamental safety finding. It means that the way your clients deploy AI, not just which AI they buy, determines the safety profile. This has direct implications for how health systems should think about AI vendor evaluation. A model that performs poorly on safety benchmarks without customization but responds well to safety prompting is a different risk category than a model that is unsafe regardless of how it's prompted. For vendors selling clinical AI tools: your safety profile should include documentation of the system prompts and guardrails you've implemented, not just benchmark performance on raw model outputs. For health systems buying those tools: you should be asking vendors to show you their safety prompt architecture, not just their benchmark results. The Mount Sinai finding also has implications for the AI governance frameworks being developed this week by the AMA and AHA. Governance frameworks that focus exclusively on model evaluation at point of procurement miss the safety levers that exist at point of deployment. Prompt engineering, system configuration, and ongoing monitoring are governance functions, not just technical functions.
Healthcare AI Weekly by Greg Harrison