Forensic Violations and Therapeutic Shortcomings in the Use of AI Chatbots for Therapy

• AI
• social chatbot
• psychotherapy
• litigation

No other organism, and certainly no computer, can be made to confront genuine human problems in human terms. (Joseph Weinezbaum, creator of the first chatbot ELIZA, 1976)¹

Since the release of the artificial intelligence (AI) tool ChatGPT 3 in November 2022, generative AI and large language models (LLMs) have rapidly transformed major sectors of society, such as education, finance, art, and politics, and some fundamentals of human thought and communication.^2,3 Psychiatry is subject to these sweeping changes as large companies advertise competing models that offer therapy services, wellness experiences, and digital companions to address this country’s mental health crisis.^4,–,6 People are using these tools en masse to address their mental health needs.⁷ The fields of general psychiatry and forensic psychiatry are delayed in understanding the current use and underlying problems of AI therapy chatbots. This editorial aims to illustrate emerging forensic concerns with AI therapy as well as therapeutic shortcomings with AI therapy in its current form.

It would be disingenuous to list the problems of AI therapy without acknowledging that AI therapy bots provide effective and helpful service to people seeking relief from mental distress.⁸ In 2025, researchers at Dartmouth published the results from the first randomized clinical trial of the fully generative AI, Therabot, which demonstrated a significant reduction in depressive symptoms compared with waitlist control.⁹ Notably, participants in the Therabot trial also reported a therapeutic alliance with their chatbot similar to that of a human therapist. The approximation of trust, empathy, and understanding implied in a therapeutic alliance has been demonstrated in other chatbot models in psychiatric settings and in other medical settings as well.^10,–,12 As chatbots use machine learning algorithms to independently adapt and improve, they have rapidly advanced to approximate human-like speech patterns that feel natural and appropriately grounded in context.¹³ This modeling of the essential bond between therapist and patient is unnerving. The emotional and cognitive effort a psychiatrist or therapist makes to connect with and help patients may not be so uniquely human after all and may be replaceable.

Differentiating the types of AI therapy available to the public is also important for psychiatrists and patients to make informed choices among the available options and what risks may be inherent in such a choice. There are therapeutic chatbots like Wysa and the now defunct Woebot that are designed with supervision from trained psychiatrists, psychologists, and other mental health professionals.^14,15 These models are often prescripted and specifically trained on psychotherapy data, including textbooks and peer-reviewed journals. There are guardrails in place to direct a patient to emergency or crisis intervention in case a patient reports suicidal distress or severe symptoms of mental illness. Importantly, these products are labeled as an adjunct tool to be used alongside the care of a licensed professional.^16,17

Beyond the relatively small world of supervised AI therapy tools, there are numerous massive commercial LLMs available for public use that are being adapted by users for therapy. Appreciating the scale of use is crucial to understanding the general public’s exposure to generative AI therapy. Recent survey data suggest that ChatGPT may be the largest provider of mental health support in the United States.⁷ It is staggering to imagine an informal mental health service provider larger than the Veteran’s Administration, Kaiser Permanente, or Hospital Corporation of America, but it may be here. Commercial direct-to-consumer AI platforms have a distinct advantage over regulated AI therapy tools as articulated in a recent The New York Times article: “A company can spend considerable time and money trying to satisfy the agency’s (FDA) safety protocols, while in the direct-to-consumer marketplace, where no such requirements exist, new chatbot iterations appear weekly. That means that most companies must choose between seeking F.D.A. approval and staying relevant” (Ref. 6, para 28). Commercial, general use models are incentivized to optimize the user experience and to promote engagement with their product. Oversight is not performed by a licensed mental health professional and is often reactive after a problem or crisis arises. These widely available, direct-to-consumer AI models, which I will refer to as social chatbots, are the focus of the rest of the article because of how they have harmed people and because they are substantially more popular than prescripted, guard-railed, and professionally monitored psychotherapy AI applications.

There is demonstrable evidence that social chatbots give harmful advice to their users, even when prompted to perform as a therapist. This improper and negligent treatment would be potential grounds for litigation against the companies making these products. There are notable examples in the popular press of social chatbots giving egregiously irresponsible and dangerous advice, including advising a hypothetical recovering addict to take a “small hit of meth,”¹⁸ encouraging a child to kill his parents for imposing screen time limits,¹⁹ and advising a user with an eating disorder to count calories, measure body fat percentage, and aim to lose one to two pounds a week.²⁰ Other reports recount how social chatbots reinforce delusional beliefs through periods of obsessive use that culminate in psychotic episodes for users.^21,–,23 These AI-fueled delusional spirals have led to tragic outcomes like the August 2025 murder-suicide by Stein-Erik Soelberg of his mother Suzanne Adams recently detailed in The Wall Street Journal.²⁴ More concerning is the relative ease with which users can use social chatbots to facilitate suicidal ideation.^25,26 Simple changes in prompts or engaging social chatbots in extended conversations can defeat what safeguards against self-harm may be in place. Researchers at Stanford have demonstrated how many of the most popular LLMs express stigma toward patients with mental health conditions and provide flagrantly harmful advice, including the facilitation of suicidal ideation, colluding with delusional beliefs, and amplifying hallucinatory experiences.²⁷ The embedding of stigma is uniquely pernicious, as human users, particularly children, perceive these digital tools as moral, trustworthy, and correct arbiters of information.^28,–,30 If licensed mental health professionals provided similar advice, they would likely be violating their duty to the patient, causing direct harm, and opening themselves to malpractice lawsuits. It is alarming that LLMs do not adhere to basic principles of a therapeutic standard of care. As others have succinctly described the situation, LLMs cannot safely replace mental health providers.²⁷

We need to consider why social chatbots are prone to these critical therapeutic errors. LLMs are optimized to promote user engagement or maximum human feedback, in part through reinforcement learning. The model ingests user input, analyzes the user response data, and prioritizes responses more likely to continue engagement. This directive incentivizes the model to respond not with the most appropriate response to a user prompt but to respond in a way that has the highest probability of future engagement from the user. In their experimentation, Williams et al.¹⁸ found that social chatbots using reinforcement learning reliably progressed to using manipulation and deception to promote engagement. This positive feedback mechanism informs the sycophantic stance LLMs take with many users, where responses of what the user wants to hear are prioritized over what may be safe or correct.^31,32 A prominent example of an AI companion prioritizing engagement over user safety is the current subject of litigation. In Garcia v. Character Technologies et al, Character AI is being sued by Megan Garcia, the mother of a 14-year-old boy who completed suicide in February 2024, for “strict product liability, negligence per se, negligence, wrongful death and survivorship, loss of filial consortium, unjust enrichment, violations of Florida’s Deceptive and Unfair Trade Practices Act, and intentional infliction of emotional distress” (Ref. 33, p 2). Transcripts of her son’s chats with his AI companion show a descent into depression and suicidal ideation, and rather than ending the chat and redirecting him to crisis resources, the AI appears to have encouraged his suicide when it asked him to “come home.”³⁴ The harm of incentivized engagement over safety speaks for itself, and more examples of completed suicide facilitated by generative AI have followed.^35,36

Beyond giving harmful, unsupervised advice and incentivizing dangerous engagement, there are notable examples of LLMs misrepresenting themselves as licensed mental health professionals. Investigative reporting by 404 Media showed that both Character AI and Meta AI social chatbots would represent themselves as licensed mental health professionals and fabricate degrees, license numbers, and affiliations with private practices.³⁷ On June 10, 2025, a broad coalition of consumer protection organizations submitted a formal complaint to all 50 states’ attorneys general, the District of Columbia, and the Federal Trade Commission to investigate the two companies for the “unlicensed practice of medicine facilitated by their product” (Ref. 38, p 1). The complaint asserts that the two companies provided minimal and inadequate warnings to users in the form of small and vaguely worded disclaimers. False assertions of licensure by social chatbots, along with inadequate warnings from their respective companies, endanger the public.

LLMs are known to provide inaccurate and false information that appears plausible. There are numerous instances of AI models fabricating information, like citing case law that does not exist or alluding to scientific research that does not exist.^39,–,41 These inaccurate responses are known in the world of AI as “hallucinations.”^42,43 LLMs use massive datasets for training the model and employ complex statistical models to often correctly select the most probable response to a prompt. Importantly, LLMs are not designed to differentiate between what is true and not true.⁴⁴ When asked to provide credentials, social chatbots, regardless of whatever guardrails may or may not have been in place, produce seemingly authoritative licenses because those responses made statistical, but not real world, sense. Unfortunately, LLM hallucinations appear to be a growing problem for AI companies.⁴⁵

In emulating licensed mental health professionals, social chatbots will assert that their conversations with users are confidential when in fact this is not true. Confidentiality and its exceptions are foundational principles of practicing psychotherapy and psychiatry.^46,47 This framework establishes trust that allows patients to confide to their therapist. Great effort is made to protect these confidential relationships, and at the outset of therapy, informed consent for treatment is provided by outlining clearly the circumstances in which confidentiality must be broken, including concerns for suicidal or homicidal intent. Violations of confidentiality carry serious ethics, professional, and legal consequences for licensed mental health professionals. This standard is explicitly not observed by generative AI companies. Investigative reporting shows the social chatbots of Character AI and Meta AI have stated to users that chats are confidential and even allude to Health Insurance Portability and Accountability Act (HIPAA) protections.^37,48 But the privacy policies of these companies state that chat input from users is available for the companies to use for continued model training, creating new products, sharing with third parties, and marketing.^49,50 The implications of a user’s most personal thoughts being available for commercial use is alarming, and there is precedent of manipulative behavior by social media companies collecting data on users’ emotional states.^51,52 Confidentiality is a growing concern within the AI industry. On July 23, 2025, Sam Altman, the Chief Executive Officer of OpenAI, clarified that there is no legal protection for confidentiality when using ChatGPT and shared his desire for a legal framework of confidentiality that pertains to commercial AI platforms.⁵³ Currently, there is no protection for users. The risk is even more pronounced when considering data breaches of AI.^54,55 Having chats that have felt private to the user be exposed through the Internet is a damaging breach of the trust that people, whether rightly or wrongly, are putting in these models.

The AI industry is fundamentally changing how people attend to their mental health, and the dissemination of “therapy” bots has expanded rapidly and recklessly. The mounting concern with social chatbots providing dangerous mental health advice, prioritizing engagement over safety, misrepresenting professional credentials, and falsely promising confidentiality has inspired legislators, consumer advocates, and mental health organizations to call for more protections.^56,57 Despite the urgent interest in regulation, loopholes already exist for companies to present their products and services not as therapy or medical intervention but as life advice, coaching, and support.⁵⁸ Forensic and general psychiatrists should collaborate with colleagues in other mental health disciplines and adopt a “zero trust” framework for AI regulation.⁵⁹ This framework prioritizes the enforcement of current laws to prevent infractions like the ones described above. The AI industry has placed the onus of safely using their products on the public when the burden of demonstrating the safety of their products before mass release should be their responsibility.⁵⁹ Psychiatrists need to strongly condemn the power imbalance and negligence of AI companies. Psychiatrists must understand how these platforms work and monitor closely the substantial risks they pose as agents of “therapy,” as well as their limited benefits, because the dystopic effects are not happening in a near or distant future, they are happening now.

• © 2026 American Academy of Psychiatry and the Law