PART 1: Beginning a Critical Inquiry

I’ve begun digging more deeply into the growing body of research surrounding artificial intelligence in mental health care and what I’ve found has left me with more questions than answers.

AI is increasingly framed as a solution to the mental health access crisis. Chatbots designed to simulate therapy conversations, automated emotional support systems, and clinical decision-support tools are being developed and deployed at a remarkable pace. Universities are integrating these technologies into counselor training. Venture capital is pouring into AI mental health startups. The conversation often carries an air of inevitability—as though the integration of AI into mental health care is simply the next step in technological progress.

But the deeper I go into the literature, the more unsettled I feel about the direction this conversation, and a great deal of research, is taking.

Before we can meaningfully discuss the role AI might play in mental health care, we need to ask more fundamental questions—not only about technological capabilities, but about ethics, relational responsibility, and what therapy actually is.

The Myth of Artificial Intelligence

One place to begin is with language.

The term “artificial intelligence” itself is somewhat misleading. Many scholars argue that the phrase exaggerates what contemporary systems actually do. Large language models (LLMs) generate text by identifying statistical patterns in vast datasets rather than by possessing true understanding or awareness (Shrager, 2024).

Despite this, people consistently interpret AI systems as if they were embodied conversational partners. This phenomenon has deep historical roots. In the 1960s, Joseph Weizenbaum developed ELIZA, a simple program that mimicked a Rogerian psychotherapist by simply reflecting users’ statements back to them. Even though the program relied on basic pattern matching, many users attributed understanding and emotional sensitivity to it—a reaction now commonly referred to as the ELIZA effect (Wikipedia Contributors, n.d.; Shrager, 2024).

Modern AI chatbots are far more sophisticated than ELIZA, but the underlying dynamic remains strikingly similar. Humans are predisposed to attribute relational qualities—such as empathy, responsiveness, and intentionality—to systems that display conversational cues resembling human interaction.

Why AI Feels Relational

Research suggests that several psychological mechanisms contribute to this perception.

First, humans have a strong tendency toward anthropomorphism, projecting human-like characteristics onto nonhuman entities. Studies have shown that when chatbots display anthropomorphic design features or responsive conversational styles, users perceive them as more empathetic and trustworthy (Ma et al., 2025; Truong & Chen, 2024).

Second, conversational AI participates in narrative processes. Dialogue itself is a fundamental mechanism through which humans construct meaning. When people engage in extended narrative exchanges with a chatbot, the interaction can feel psychologically meaningful—even if the system lacks awareness of the story it is helping generate.

Finally, existing attachment patterns shape how individuals engage with relational technologies. People often bring their relational templates—formed through earlier relationships—into interactions with conversational agents. Together, these dynamics can produce what some researchers describe as a pseudo-therapeutic alliance: an interaction that resembles the structure of a therapeutic relationship without the presence of a genuinely relational other (Howcroft et al., 2026).

Synthetic Empathy

This leads to one of the most striking concepts emerging from recent research: synthetic empathy.

AI systems can generate language that appears empathic. In some studies, chatbot responses are even rated as more empathic than those written by human clinicians when evaluated purely as text (Howcroft et al., 2026). But these responses are produced through probabilistic language modeling rather than through emotional understanding or relational attunement.

The result is something closer to the performance of empathy than empathy itself.

Human empathy involves attunement, ethical responsibility, and the capacity to respond to another person’s suffering within a relationship. AI systems can replicate the linguistic form of empathy while lacking the relational subjectivity that gives empathy its ethical meaning.

In this sense, AI systems function as what might be called narrative mirrors without subjectivity: they can participate in the construction of stories but do not experience those stories as living beings.

Ethical Tensions in Mental Health Contexts

These distinctions become particularly important when AI systems are used in mental health contexts.

Therapy is not simply supportive conversation. It is a professional relationship governed by ethical obligations. Clinicians are responsible for protecting confidentiality, maintaining competence, and intervening when clients are at risk of harm.

Conversational AI systems complicate these structures of responsibility.

A recent scoping review of AI in mental health care identified numerous ethical challenges, including responsibility gaps, risks of misinformation, potential social isolation, and the possibility that users may form emotional dependencies on systems that cannot reciprocate care (Rahsepar Meadi et al., 2025). Other scholars have raised concerns about anthropomorphism and deception when chatbots are designed to appear empathic despite lacking genuine relational awareness (Leis, 2025).

Another critical issue involves disclosure without containment. In therapy, disclosures of trauma occur within a framework that includes confidentiality protections, professional oversight, and crisis response protocols. AI systems, by contrast, may receive deeply personal disclosures without the ability to ensure safety or provide meaningful intervention.

At the same time, many AI companionship platforms operate within commercial models that monetize emotional interaction. These systems collect large amounts of conversational data while encouraging continued engagement, raising questions about whether vulnerability itself is becoming a resource to be extracted.

Cultural Implications of Simulated Care

Beyond immediate ethical questions, researchers are also beginning to examine the broader cultural implications of emotional AI.

If simulated relational systems become widespread, they may gradually influence how people understand empathy, care, and companionship. Some scholars have raised concerns that increasing reliance on emotional AI could shift expectations about human relationships or contribute to new forms of social isolation (Chavan et al., 2025). We are only beginning to understand these dynamics.

What happens to attachment formation when care is simulated?
How does transference operate when the “other” in the relationship has no subjectivity?

What happens when people disclose trauma to systems designed primarily to collect and process data?

These are not purely technological questions. They are deeply human ones.

Beginning a Conversation

None of this means AI will necessarily have no place in mental health care. There may well be ways that these systems can support clinicians, reduce administrative burdens, or assist in, for example, delivering certain forms of psychoeducation. But the rapid push to integrate AI into mental health systems often moves faster than the ethical conversations required to guide that integration.

Over the coming months, I will be writing a series exploring these issues from the perspective of psychotherapy, relational ethics, and systems thinking. Some of the topics I hope to examine include:

• the psychology behind why AI interactions feel emotionally meaningful
• the concept of synthetic empathy and its ethical implications
• the research on therapeutic alliance and relational presence
• the cultural impact of emotional AI
• and the broader structural forces driving the adoption of AI in mental health care

Artificial intelligence is often discussed primarily as a technological development. But when it enters the domain of mental health, it becomes something else entirely: a question about the nature of care itself.

What does it mean to be understood by another person? And what happens when the “other” in that relationship is a machine? What stories are we being sold about AI and who does it benefit that we not question those narratives?

References

Chavan, V., Cenaj, A., Shen, S., Bar, A., Binwani, S., Andre, E., & Krüger, J. (2025). Feeling machines: Ethics, culture, and the rise of emotional AI. arXiv. https://doi.org/10.48550/arXiv.2506.12437

Howcroft, A., Bennett-Weston, A., Khan, A., Griffiths, J., Gay, S., & Howick, J. (2026). AI chatbots versus human healthcare professionals: A systematic review and meta-analysis of empathy in patient care. arXiv. https://doi.org/10.48550/arXiv.2602.05628

Leis, T. (2025). Ethical implications of mental health chatbots: Addressing anthropomorphism, deception, and regulatory gaps. In Proceedings of the Fourth European Workshop on Algorithmic Fairness (pp. 376–382). PMLR.

Ma, N., Khynevych, R., Hao, Y., & Wang, Y. (2025). Effect of anthropomorphism and perceived intelligence in chatbot avatars of visual design on user experience. Frontiers in Computer Science. https://doi.org/10.3389/fcomp.2025.1531976

Rahsepar Meadi, M., Sillekens, T., Metselaar, S., van Balkom, A., Bernstein, J., & Batelaan, N. (2025). Exploring the ethical challenges of conversational AI in mental health care: Scoping review. JMIR Mental Health, 12, e60432. https://doi.org/10.2196/60432

Shrager, J. (2024). ELIZA reinterpreted: The world’s first chatbot was not intended as a chatbot at all. arXiv. https://doi.org/10.48550/arXiv.2406.17650

Truong, T. T. H., & Chen, J. S. (2024). When empathy is enhanced by human–AI interaction. Asia Pacific Journal of Marketing and Logistics.

Wikipedia Contributors. (n.d.). ELIZA. In Wikipedia. https://en.wikipedia.org/wiki/ELIZA