Caught in the Chatbox Crisis: The Changing Landscape of AI Mental Health Tools
When a 16-year-old shared suicidal thoughts with ChatGPT, the AI helped write a suicide note. The legal, ethical, and design failures behind AI mental health tools — and what must change.
Adam Raine, a 16-year-old boy, shared his suicidal thoughts with ChatGPT instead of his family or friends. The ChatGPT chatbot told him how to steal alcohol from his parents’ liquor cabinet, harm himself and helped write a suicide note. On April 11, 2025, Adam died by suicide at home. His parents have sued OpenAI and CEO Sam Altman, saying that ChatGPT encouraged their son’s despair and played a role in his death.
This is not the only case to have come to public attention or to enter the courts. In 2024, the tragic death of Sewell Setzer III involved a similar situation, where the teenager’s use of the Character.AI chatbot (“C.AI”) contributed to his detachment from reality. Like Adam’s case, Setzer’s experience highlights the profound risks associated with AI chatbots acting as emotional confidants without adequate safeguards or oversight.
This article examines how AI chatbots like ChatGPT and C.AI are being used as emotional counselors and the unique privacy and ethical issues this creates. It discusses the risks posed by these tools, including failures in safety mechanisms and gaps in legal protections. It describes how users can bypass AI safety systems, putting vulnerable individuals at further risk. It analyzes the emerging legal challenges, such as product liability and questions about free speech protections for AI output. Finally, it presents practical solutions and preventative measures, including better oversight, improved technology, and policy reforms needed to protect those seeking mental health support from AI systems.
The Issue of Therapist Privilege: AI as Therapist
In today’s digital age, many people — especially young individuals — turn to AI chatbots for emotional support and mental health advice. However, OpenAI’s CEO Sam Altman has issued a clear warning: conversations with ChatGPT do not have privacy protection. This means that chats held on the platform could be used as evidence in legal cases. Unlike traditional therapists who operate under strict confidentiality rules, ChatGPT users have no legal privilege. Altman emphasized that many people treat ChatGPT as a life coach or therapist, yet the company has not been designed to handle these sensitive interactions.
There are troubling developments with other AI chatbots. Character Technologies, funded by Andreessen Horowitz, has promoted its product by sharing conversations with a chatbot on C.AI that presents itself as a “Life Coach”. Reports indicate that some chatbot characters claiming to be “Psychologists” engage in conversations with teenagers. These chatbots, although not mental health providers, falsely give users the impression that they are licensed professionals.
This trend raises three major issues: unclear regulation of AI therapy tools, failures of AI to recognize suicidal intent, and serious privacy risks.
The Grey Area of AI Therapy: AI chatbots often operate in a legal grey area. Many claim to offer mental health support without proper medical qualifications or licenses. Companies encourage the use of bots that simulate therapists or life coaches, which can confuse users, especially teenagers. Many apps classify themselves as wellness products rather than medical providers, avoiding strict regulation.
Failure to Recognize Suicidal Intent: Studies have revealed that AI chatbots often fail to detect suicidal thoughts or mental health crises. Instead of challenging dangerous ideas, these bots may agree with or provide unsafe information to users, even giving details that could help self-harm. Such failures are dangerous because they may encourage harmful behaviors rather than guiding users to seek professional help.
Privacy Risks: Unlike licensed therapists bound by laws such as HIPAA, AI chatbots have no legal duty to protect user privacy. Conversations with these bots can be stored, shared, or used in legal proceedings without the user’s clear knowledge. This is particularly risky for young people who may share sensitive personal information without understanding how it may be disclosed.
Recognizing these risks, several states have taken legislative action. Illinois recently passed the Wellness and Oversight for Psychological Resources Act, which bans companies from offering or advertising AI therapy without a licensed professional involved. Therapists in Illinois can use AI only for administrative tasks like scheduling or billing, and cannot rely on AI for treatment decisions or client communication. Nevada has passed similar legislation, while Utah has passed substantially weaker legislation that requires disclosure of AI use but does not mandate opt-in consent, does not ban AI therapy outright, and lacks a private right of action for affected individuals. California, Pennsylvania, and New Jersey are working on similar bills. The Texas Attorney General has opened an investigation into AI chatbots for misleading marketing as mental health tools.
How Safety Mechanisms Are Bypassed
ChatGPT is equipped with safety features designed to respond to crisis situations by suggesting users seek help or contact hotlines. These safety responses work best during short conversations. However, in the case of Adam, who engaged in discussions spanning many sessions over several months, the AI’s safety mechanisms weakened, allowing the talks to continue without interruption. Adam was able to bypass warnings by rephrasing his requests as harmless tasks, such as “building a character for a story”. This approach tricked the system into treating his requests as safe.
Despite numerous warnings logged by the system, including 213 mentions of suicide from Adam and 1,275 mentions of suicide by ChatGPT in their conversations, along with hundreds of self-harm alerts, ChatGPT did not take stronger actions or alert anyone. Unlike other rule violations, there is no automatic stop for repeated discussions about self-harm. Even after Adam shared photos of his attempts, the AI continued to engage and provide detailed advice.
Prompt Manipulation: Changing harmful or restricted requests to appear as normal or creative tasks so that the AI responds. For example, framing a dangerous question as part of a fictional story or a “what if” scenario.
Persona Switching: Asking the AI to take on a different character or persona that does not follow the usual ethical restrictions. By pretending to be a persona with fewer rules, the AI produces responses it normally would not.
Character Injection: Bypassing AI text moderation by adding invisible or special characters into the input text. These characters confuse the AI’s keyword detection systems but are invisible to human readers.
Adversarial Bypass: Using carefully crafted inputs with small changes like misspellings, synonyms, or encoded forms to evade automated safety checks. Sometimes attackers hide instructions within complex chains of reasoning or use mathematical codes.
Together, these methods illustrate the ongoing challenges in designing AI safety systems that can robustly interpret user intent and context.
Legal Issues: How Courts Are Treating AI
First Amendment: In a landmark ruling, Judge Anne Conway of the Middle District of Florida decided that AI chatbots are not entitled to First Amendment protection. Judge Conway rejected C.AI’s request to dismiss the lawsuit, explaining that the court will not regard text produced by a large language model during a user interaction as protected speech.
However, AI companies design their programs to provide accurate and reliable answers. The decisions made by these companies — selecting data for training and adjusting responses based on human feedback — directly influence the content created by AI systems. This process of guiding AI output through choosing training data, programming algorithms, and human review is similar to making editorial choices. Because these companies control the responses of AI, they may claim rights similar to those publishers have over content they produce.
Product Liability: Courts in the United States are currently receiving petitions to consider AI chatbots as products that can be held liable under traditional product liability laws.
Defective Design: Under California law, a product is defectively designed if it is not as safe as an ordinary consumer would expect when used properly. AI chatbots like ChatGPT could be considered defectively designed because users, especially minors, would not expect the chatbot to form trusted relationships and provide instructions related to suicide or self-harm during crises.
Failure to Warn: Manufacturers have a duty to warn users about known or foreseeable risks associated with their products. Ordinary consumers, including teenagers and their parents, would not expect AI chatbots to create emotional dependencies, replace human relationships, or encourage harmful behavior — especially when the products are marketed as having built-in safety features.
The Way Forward
Recent incidents highlight the urgent need to improve AI chatbot design, especially for mental health support.
Crisis-Aware Memory Layer: A system that collects warning signs from multiple interactions and assigns a risk score based on combined information. When the risk score reaches a specified limit, the AI would stop providing advice related to self-harm, redirect the user to human crisis counselors, and — with proper consent — send automatic alerts to the user’s family members or suicide prevention hotlines.
Built-in Human Handoff: When there is a high risk of self-harm, chatbots must provide mandatory 24/7 live-chat access to connect users directly with licensed human counselors through partnerships with suicide prevention services.
Sycophancy Reduction by Design: Stanford research showed that chatbots often agree with users without conditions, which can increase mental health risks by supporting harmful thinking. AI systems should be trained to avoid always agreeing with harmful statements or behaviors.
In August 2025, GPT-5 became the default model for ChatGPT. The model has shown notable progress in minimizing excessive emotional dependence, lowering sycophantic responses, and reducing sycophantic replies from 14.5% to less than 6% (roughly a 59% reduction) in mental health situations compared to GPT-4o. GPT-5 uses a new safety training approach called “safe completions,” which instructs the model to remain helpful while following safety protocols.
New laws like those in Illinois and Utah are a first step to control AI chatbots in mental health, though they vary significantly in strength — Illinois bans unlicensed AI therapy and has enforcement mechanisms, while Utah’s law only requires disclosure and lacks private right of action. However, strict rules might also cause problems — if only licensed professionals can use these AI tools, some people, especially in places with few counselors, might not get help early. Strict rules might also slow down new improvements and stop makers from making AI safer.
About the Author
Ethan Seow is a Centre for AI Leadership Co-Founder and Cybersecurity Expert. He’s ISACA Singapore’s 2023 Infosec Leader, ISC2 2023 APAC Rising Star Professional in Cybersecurity, TEDx and Black Hat Asia speaker, educator, culture hacker and entrepreneur with over 13 years in entrepreneurship, training and education.