New NEJM Catalyst case study outlines a clinician-in-the-loop model for patient-facing generative AI, with stronger safeguards, clear escalation paths, and continuous clinical oversight
SAN FRANCISCO–(BUSINESS WIRE)–Included Health, a new kind of healthcare company delivering personalized all-in-one healthcare to millions of people nationwide, today announced the publication of a new peer-reviewed paper in NEJM Catalyst Innovations in Care Delivery outlining a practical framework for deploying safe, patient-facing generative AI in healthcare.
The paper details how Included Health developed and piloted a clinically governed AI assistant designed to provide safe, accurate health guidance for routine questions while escalating higher-risk situations to qualified clinician support. At a time when more people are turning to public AI tools for quick medical answers, the paper makes the case for a different model in healthcare: one built to recognize risk and communicate limitations clearly, with appropriate human oversight.
“At Included Health, we believe healthcare AI should be held to a higher standard than general-purpose tools. People deserve timely support, but they also need a model they can trust to provide accurate, safe guidance,” said Ami Parekh, MD, chief health officer at Included Health. “That means building for the realities of care from the start — being clear about what AI should and should not do, recognizing when risk is higher, and making sure clinical expertise is part of the experience when it matters most.”
The model reflects Included Health’s broader AI + EQ approach and is built around four core principles:
- Cross-functional governance with executive sponsorship and clinical oversight from the start
- Proactive risk analysis and pre-launch testing to identify failure modes and stress-test guardrails
- A three-tier risk classification system for standard, high-risk, and emergency questions
- Continuous clinician-in-the-loop review to audit interactions and improve the system over time
During the seven-week pilot, Included Health deployed the risk-stratified assistant to 50% of its patient population and manually reviewed 100% of clinical interactions daily. Clinical review found the approach maintained a high level of safety, and reduced unnecessary handoffs for standard-risk questions by 65%, giving more members instant and clinically-appropriate guidance for their healthcare needs. Patient experience also remained high and on par with the comparison group.
“We believe the framework described in this paper offers a practical blueprint for how healthcare organizations can deploy patient-facing AI with safety, transparency, and human oversight built in from the start,” said Ankoor Shah, MD, vice president of clinical excellence at Included Health. “That is the standard this industry should be building toward — and the kind of governance that can help raise the bar for how these tools are deployed and trusted across healthcare.”
The full paper, “Blueprint for Safety: Implementing a Clinically Governed AI Digital Assistant for Patient Guidance,” appears in the July 2026 issue of NEJM Catalyst Innovations in Care Delivery.
About Included Health
Included Health is a new kind of healthcare company that delivers personalized all-in-one healthcare to millions of people nationwide. We provide healthcare access, answers, and advocacy through a modern experience designed to treat people better — mind, body, and wallet. It’s all included: comprehensive virtual and in-person care, system-wide navigation and care coordination, and 24/7 support for every clinical and administrative need. Our members see better doctors, experience better outcomes, and have more healthy days, all at lower cost. Learn more at includedhealth.com.
Contacts
Media Contact
Alex Maynard
[email protected]
