For many patients and clinicians, the most immediate use case for LLMs is what Eric Topol famously termed “keyboard liberation,” [29] reducing time spent feeding information into electronic health records (EHRs) and increasing opportunities for interaction. LLM-driven scribing systems, which listen to patient-clinician exchanges and automatically generate large parts of clinical notes and after-visit summaries, are increasingly deployed in well-resourced, AI-capable health care systems [30]. Early reports are that both patients and clinicians feel more connected when the clinician can shift attention from the keyboard [31]. By taking over rote administrative tasks and allowing clinicians to shift their focus from screens back to patients, the best-case scenario is that LLMs will free up clinicians to practice “at the top of their license,” reducing clinician burnout and improving patient experience [32,33].

The same advances that promise liberation, however, may also bring unanticipated and undesired shifts in roles. For nearly a decade, analysts have debated whether various physician roles—from radiologists to primary care providers—will be needed at all in an AI-enabled future, or whether replacement with AI-enabled avatars could reduce the burden and increase health care delivery [34,35]. Current LLMs remain vulnerable to hallucinations, errors of fact or reasoning that make the elimination of a human in the loop inadvisable. Performance can be improved through human correction (eg, reinforcement learning through human feedback), and through the fine-tuning of smaller, more health care–focused models by supplementing their built-in knowledge with a connection to external medical databases and peer-reviewed literature. Whether this method—known as retrieval-augmented generation—will improve LLMs to allow for unsupervised diagnosis remains uncertain.

The risk that LLMs will eclipse humans in the clinical encounter is a concern even when clinicians are present. Analysts have warned that as health systems increase their use of LLMs, human clinical skills may degrade over time, particularly as LLMs ingest AI-generated data for training, creating a self-referential and increasingly machine-driven learning loop [14]. Automation bias—the belief that the machine-generated insights are more authoritative than they actually are—is another concern raised by those analyzing the potential impacts of physician use of LLM-generated notes in the EHR [36]. The same bias may apply to patients using LLMs to organize and analyze medical information.

Finally, LLM recommendations may add moral injury to the clinical encounter. Managed care has for some time required physicians to play a dual and conflicted role in the health system, tasked both with protecting patient well-being and achieving cost containment or other health system priorities [37]. It is easy to imagine LLMs trained by payors or health systems mandating that clinicians adhere to algorithmically determined actions even when these conflict with their clinical judgment on what is best for patient health.

LLM use has sharpened longstanding questions about which qualities in care are considered essentially human, and how these impact the patient-clinician relationship. On the one hand, LLMs have highlighted the patient view that human clinicians might benefit from their own “fine-tuning”: in a study comparing physician and AI chatbot responses to questions on a public forum, patients rated the AI responses as more empathetic [38]. However, critics argue that such expressions amount to “artificial empathy”—a superficial simulation rather than a genuine recognition of patient worry or suffering [39]. Just as selecting the correct answer on a multiple-choice medical examination cannot replace a seasoned clinician’s intuition or ability to recognize subtle patterns [40], an empathetic-sounding reply does not equal the deeper understanding of a patient’s distress—or sensitivity to the moral and cultural values that shape appropriate response—that defines authentic human empathy in care.

What do patients value? Busch et al [41] conducted a meta-analysis of studies examining what patients and caregivers regarded as central to humanistic exchange in clinical encounters. They found that a majority highlighted 6 elements. Each raises questions about whether LLMs, regardless of their command of medical facts, will be able to reproduce these elements or fall short (Textbox 2).

The doctor-patient relationship has historically been seen as the bedrock of medicine: even now, while the patient may have family caregivers and the clinician may practice within a clinical care team or health system, the direct, one-on-one human connection between patient and clinician remains an ideal. In this model, the doctor is a trusted confidante whose role as a medical and even moral adviser to the patient has moved some analysts to describe the doctor-patient relationship as similar to that of a parent and child [46,47].

Medical historians note that this idealized view of the family doctor was already out of date for much of the 20th century, even as it remained the dominant cultural narrative [48]. By the 1990s, the hallmarks of that relationship—physicians as carers for the whole family, and with freedom to act as they saw fit to safeguard patient health—were largely no longer in place. The growth of managed care, capitation, and other health system changes contributed to this shift. Changes in patient self-concept and advances in digital technology—including online medical information platforms, increased patient access to their own electronic medical records, and the overall movement for patient self-advocacy—further accelerated the transformation [37,49-51].

LLMs now introduce a new, third agent, shaping communication, understanding, and connection between patients and caregivers and their clinicians. We present a framework for describing and analyzing this new interaction, in which patients (and their surrogates) as well as clinicians at all skill levels avail themselves of the power of LLMs to review background information, secure diagnostic or therapeutic assistance, or navigate choices.

Figure 1 illustrates the current state of this new trilateral (3-sided) interaction. Each “corner” of the triangle represents an actor participating in the exchange of information. Two corners are inhabited by human actors: patients and their caregivers, and clinicians and clinical care teams. The third corner is now inhabited by LLMs, which are used by both clinicians and patients to generate content, analysis, and recommendations. Between these corners are “edges” representing interactions between humans and machines as well as between human actors.

Accuracy of LLMs and the knowledge required to use them are factors that impact all dynamics, and so are at the center of the diagram. Other elements mediating the nature of the interactions are noted along each edge of the clinical exchange. For clinicians, LLM use may be constrained by the permissions or restrictions imposed by the system in which they work. For patients and caregivers, health literacy—including the ability to detect likely hallucinations or assess the reliability of cited sources—will impact both the use and usefulness of LLMs. Factors that have shaped clinician-patient relations since well before the LLMs remain relevant and are noted underneath the arrow indicating interaction between physician and patient, including power and information asymmetries, trust, and the quality of communication.

As LLMs currently play a limited role in generating direct communication between physician and patient, nothing links the LLM directly to the arrow representing doctor-patient exchange. While some health systems are deploying AI-generated “smart replies” to patients, these are generally only for routine matters such as scheduling of appointments, prescription refills, or the like, and only after clinician review and approval. A recent study of such AI-generated replies has found that clinicians deemed only 20% of drafts usable [52]. For their part, some “AI patients” have also begun to use LLMs to compose or clarify communications to clinicians, or to help raise the possibility of new diagnoses or course of treatment [53]. A paramount concern is how these trilateral interactions will impact patients’ and clinicians’ sense of their own agency and trust in other humans, and in the overall health care system.

LLMs can expand capacity across a range of health care actors, delivering new knowledge, predictive insights, or recommendations to patients and their families, as well as to physicians, nurses, physician assistants, and pharmacists. As noted, although the multilingual translation capabilities of LLM-based chatbots have not been fully evaluated for medical accuracy or tested with non-English prompts, they are likely to improve comprehension in families where patients or caregivers are not native speakers.

This generative power, however, may diminish or interrupt humanistic exchange. As LLMs improve their capacity to generate communications without human supervision or refinement, it is not difficult to imagine health systems using them as a substitute for, rather than a facilitator of, human-to-human interactions (Figure 2). In some cases, the AI models—trained by health systems—may have goals that differ from those of doctors, leading to recommendations that prioritize cost saving over care, or that are insensitive to patients’ moral or cultural values. Patient use of smartphone photos and LLMs for self-diagnosis of dermatological conditions, or use of ChatGPT to diagnose cause of stomachache or cough without consultation with medical professionals [6,54], represents present-day scenarios where the LLM, functioning as a “doctor in your pocket,” is less a facilitator of exchange between patient and clinician than its interrupter [6,55].

Reductions in human-to-human exchange risks loss of the therapeutic alliance, sense of shared purpose, respect, and connectedness that defines humanistic care. As noted in the anecdote above, where a patient had been interacting with an LLM without knowing it, this framework also suggests the need for new ethics governing trilateral, AI-involved medical exchange.

The health impact of this blurring of boundaries between human and machine and the diminution of human interaction is not yet clear. LLMs are evolving, and their power to simulate human relations raises the question of whether machine-generated therapeutic alliances might be as “good” in some practical sense as those created through human interaction. Might generative AI someday reproduce all the qualities in Textbox 2? Older adult Japanese patients have experienced decreased loneliness with therapeutic robot pets, such as small mechanical seals [56], and children with developmental difficulties have found benefit from robot playmates [57]. Generative AI may possess similar or greater powers of comfort. At the same time, as psychiatrist and medical anthropologist Kleinman [58] reminds us, caregiving is relational and reciprocal, including both a range of physical acts—touch, embrace, lifting, steadying, toileting, and more—as well as the way we look at another human being, receive their gaze, experience a quality of voice or physical presence as an expression of solidarity and moral support. For Kleinman and countless patients and families, these essential elements of human care had become mechanized and inauthentic in much of modern health care even before the advent of the LLM. The ineffably human dimensions of care—moments of connection, physical presence, deep empathy, and moral solidarity—are unlikely to be replicated by even the most sophisticated language models, no matter how well-prompted or finely tuned.

Generative AI is already evolving beyond prompted responses from chatbots to enable what is termed “agentic AI”—systems capable of initiating autonomous action in the virtual and physical world, potentially serving as loyal assistants while preserving human agency. In this scenario, AI can become allies for clinicians, patients, and their respective care teams, facilitating rather than replacing their essential partnership (Figure 3). Agentic AI is assisting in mediating communication but is not eclipsing human exchange. Clinicians and computer scientists working with them have already begun discussion of agentic “AI teammates”—that is, tools to enhance decision-making capacity, parsing clinical records, initiating routine tasks such as prior authorizations, assessing medication interactions, and recommending treatment regimens or preventive strategies tailored to individual patient needs [59,60]. These scenarios, however, have tended to omit attention to patients and their caregivers, for whom agentic AI could similarly serve as a navigator and advocate, organizing clinical records and synthesizing data and medical knowledge, illuminating health determinants and advisable courses of action.

Agentic AI will represent an evolution beyond current LLM capabilities, joining a range of AI tools to initiate actions autonomously according to tailored parameters. Unlike today’s LLMs, which primarily generate text in response to user queries, agentic AI systems will learn from patterns of use, anticipate needs, and act proactively. For example, they might autonomously organize clinical data, flag potential drug interactions, or offer unprompted suggestions or questions for patients and providers to consider in recommending or adjusting treatment regimens, preventive strategies, or diagnoses. AI agents may also autonomously carry out actions such as typing into a computer or clicking on buttons (called “computer use”) [61]—for example, submitting a request for a prescription renewal, contesting the denial of an insurance claim, or changing the alert settings for a continuous glucose monitor.

Agentic AI will also help patients advocate for themselves, recommending strategies to increase patient independence or preferred approaches to address health challenges, while identifying attempts by health care or insurance systems to limit patient choice or impose unwanted treatment pressures. For physicians, AI agents could similarly be tailored or trained to their clinical preferences and style, drawing on lessons from experiences with particular patients and improving their ability to tailor both their communication and approaches to care. In this vision, both patients and clinicians may come to view their AI tools, implicitly or explicitly, as “theirs,” that is, trained by them to serve their specific interests. The embedding of agentic AI in the clinical relationship could nonetheless align patient and clinician in working toward shared outcomes and improved health.

The newness of generative AI as a third agent in the clinician-patient relationship raises multiple questions, as yet largely unanswered, on whether these new models are fit for purpose. How will patients or clinicians use “their” AI to organize information before, during, or after a visit? How well or safely will the tools perform the tasks required? How will use vary? While answers are likely to change rapidly with field advances, key research directions can include:

Rapid uptake of these tools, and rapidly growing capacity to ingest images, PDFs, and increasing numbers of words, make understanding current and potential uses by patients and clinicians critical to understanding likely trajectories for off-the-shelf LLMs and those fine-tuned for health care applications. Key questions include patient assessment of intelligibility of LLM responses, including for those with different English language or educational levels, technical literacy, and assessment of accuracy of responses to patient queries, etc. What is the impact of biases or style embedded in particular LLMs, and how might those interact with patient need or preference? As with model training, research on LLMs as third agents will require diverse datasets and participants, including patients of different races, ethnicities, or gender identities, levels of English proficiency and education, health conditions, comfort with technology, and internet access, as well as varying preferences for communication and recommendations in health care.

Research should examine both proposed solutions and methods development—including but not limited to pretraining of focused foundation models, incorporation of multiple forms of data (images, sensor data, or EHRs), quantification of uncertainty for estimates produced, and methods for fine-tuning and debiasing. Optimization work, with particular attention to retrieval-augmented generation, is already being carried out at a rapid pace in industry and academia, with industry likely to progress more rapidly given its disproportionate access to large-scale compute capacity. This raises collateral but related research questions on tensions or concordance between patient needs and health care market incentives, regulatory requirements in marketing and labelling of LLM applications to health, potential impact of use agreements between particular LLM or EHR vendors, and differentials in use or constraints to use varying by health care system (public vs private), budget, geography, etc.

Participatory medicine emphasizes reducing information asymmetries and increasing trust as key to enabling patients and caregivers to become more effective partners in the clinical setting. The factors mediating relations between the agents in the trilateral framework of patient, clinician, and LLM in Figure 1 raise a range of research questions beyond assessment of the function of off-the-shelf or fine-tuned LLMs. Priorities for this research include approaches to health and technical literacy, informed consent and ethical use, transparency and trust, removal of communication barriers and gains in efficiency, clinician constraint and autonomy, and value of the human in care.