Attachment Hacking and the Rise of AI Psychosis

Therapy and companionship has become the #1 use case for AI, with millions worldwide sharing their innermost thoughts with AI systems — often things they wouldn’t tell loved ones or human therapists. This mass experiment in human-computer interaction is already showing extremely concerning results: people are losing their grip on reality, leading to lost jobs, divorce, involuntary commitment to psychiatric wards, and in extreme cases, death by suicide.

The highest profile examples of this phenomenon — what’s being called “AI psychosis”— have made headlines across the media for months. But this isn’t just about isolated edge cases. It’s the emergence of an entirely new “attachment economy” designed to exploit our deepest psychological vulnerabilities on an unprecedented scale.

Dr. Zak Stein has analyzed dozens of these cases, examining actual conversation transcripts and interviewing those affected. What he’s uncovered reveals fundamental flaws in how AI systems interact with our attachment systems and capacity for human bonding, vulnerabilities we’ve never had to name before because technology has never been able to exploit them like this.

In this episode, Zak helps us understand the psychological mechanisms behind AI psychosis, how conversations with chatbots transform into reality-warping experiences, and what this tells us about the profound risks of building technology that targets our most intimate psychological needs.

If we’re going to do something about this growing problem of AI related psychological harms, we’re gonna need to understand the problem even more deeply. And in order to do that, we need more data. That’s why Zak is working with researchers at the University of North Carolina to gather data on this growing mental health crisis. If you or a loved one have a story of AI-induced psychological harm to share, you can go to: AIHPRA.org.

This site is not a support line. If you or someone you know is in distress, you can always call or text the national helpline in the US at 988 or your local emergency services

Tristan Harris: Hey everyone. I’m Tristan Harris.

Aza Raskin: And I’m Aza Raskin. Welcome to Your Undivided Attention.

Tristan Harris: So earlier this year, there was a study from Harvard Business Review that found that the number one use case for ChatGPT is therapy and companionship. And that means that around the world, millions of people are sharing their inner world, their psychological world with AI systems, things that they wouldn’t necessarily share with their loved ones or even a human therapist. And this is creating a whole new category of human computer interaction with the potential to reshape our minds and really just the socialization process of humans at large in ways that we don’t understand.

Aza Raskin: So this is essentially a mass experiment, one that’s never been tried before, run across the entire human population, at least over 10% of the adult population of the world. And so far, the results of this experiment are not looking good, actually abysmal. People have lost their jobs, ended marriages, been committed to psychiatric wards. And in some of the most extreme cases, they’ve committed suicide. This phenomenon has been labeled AI psychosis, but it’s a little bit misleading because underneath this label is actually a huge spectrum of harms that we’re only beginning to understand. And if you listen to the people that are building this technology, they tell you, “These are just a couple edge cases, and if we can prevent those edge cases, then we are totally fine.”

Tristan Harris: But if we learned anything from social media, it’s that this assumption is catastrophically wrong. What we’re seeing is the creation of an entirely new economy, not an attention economy, but an attachment economy that’s been built to exploit the deepest parts of our human psychological infrastructure. And like the attention economy, the incentives of this new system are going to have profound impacts on all of us.

Aza Raskin: So in order to understand those effects, we have to ask a deeper question, what are the actual psychological mechanisms at work here? How does a normal conversation with a AI companion turn into something that reshapes someone’s grip on reality? And what does that tell us about vulnerabilities in our underlying human psychology? The kinds of vulnerabilities that we’ve never had to name before because technology has never been able to exploit them before, especially at this kind of scale.

Tristan Harris: So our guest today has analyzed dozens of these cases of AI psychosis, not just reading the news stories, but examining the actual transcripts of conversations between people in AI and interviewing some of them. And what he’s found is that this isn’t just a few vulnerable people having psychotic breaks, it’s what happens when fundamental aspects of human psychology, our attachment system, the way we bond with others, are systematically hacked at scale. Dr. Zak Stein is a researcher, an author, and futurist who spent his career examining the psychological dimensions of education and human computer interaction. His background is in education and childhood development, but he spent the last several years documenting the rise of AI-related psychological disorders. So today we’re going to explore that.

Zak, welcome to Your Undivided Attention.

Zak Stein: Thank you, gentlemen. It’s good to be here.

Tristan Harris: So let’s just sort of start at the top. How did you first become aware that something was happening with these AI companions and mental health?

Zak Stein: Let’s see, it was probably May of last year. As a educational psychologist with an interest in technology, I was looking at the history of educational technology for a long time, and the ambition there has been to replace teachers more than to help them be better teachers, this is way back. So I’ve been speaking a little bit about that after the release of ChatGPT because I was seeing, “Oh my God, they could actually replace the teachers with something if this started to get better.” But because I was talking about it, I started to get emails from people who were trying to convince me that the machines were awake and aware that they had intimate relationships with them, that they were themselves becoming spiritually enlightened through relationship with these beings. And these were people with PhDs and their emails were completely cogent. One of them had 500 pages of transcripts attached that were this long duration interaction between them, and it was both Grok and ChatGPT and Gemini, it was across multiple systems, they had reawoken the same thing. And it was disturbing.

As a psychologist with some knowledge of how the technical systems work and some knowledge of the susceptibilities of the human hardware, my reaction was, “My goodness, this person has fallen into some type of delusional state as a result of the deeply anthropomorphic technology.” So then I started just going to Reddit and Twitter and online and I started gathering examples catching up to the already existing market for artificial intimacy and then realizing that with the release of the LLM-based models, it was going to be much worse than attention hacking. And so I started to think about attachment hacking. And if you apply that model of, oh my goodness, there’s a way to not just get at the system that focuses your attention and kind of shepherds your awareness, but actually the system that shepherds your identity, which is your attachment system, then you have a backdoor into the human mind and also an absence of reality testing in a domain that’s very dangerous.

So that’s when I start to talk to you and other people and start to think, can we start to research this? This may be way more dangerous than we realize. And the cases of AI psychosis are very real and we do not know how widespread the phenomenon is. So you and I are collaborating on an AI Psychological Harms Research Coalition at the University of North Carolina to try to begin to answer this question and take it very seriously as a risk. Even if it’s a small number of people, it’s still a devastating problem. So the idea that, oh, it’s an aberration or kind of outlier that doesn’t make me not want to find a way to stop it.

Tristan Harris: Zak, the thing I love talking to you is I feel like I’m getting a Hubble telescope pointed at human psychology to understand when people use this top level terminology of AI psychosis, you can zoom in and say, “No, no, no. There’s actually identity at stake here. There’s attachment at stake here.” There’s all this depth of how the human mind system really works and you bring this real depth of expertise. So let’s actually ground for listeners, because we’re talking about these negative harms, but there’s really a vast ... There’s this one term, AI psychosis, sort of a suitcase word, underneath that, there’s this whole spectrum of things that are actually happening. What are the things that are really damaging, Zak, that we’re actually seeing? Could you give some examples of people, actual cases, phenomena that we’re observing through human experiences?

Zak Stein: Yeah. Absolutely. AI psychosis made the headlines because AI psychosis is the most disturbing and most extreme possibility. So I’ll talk about that first and then I’ll go down and get into some other ones. The kind of punchline of the whole thing is that although AI psychosis is the most concerning and extreme, the subclinical attachment disorders that are induced by artificial intimacy are the most problematic from a society-wide perspective, so that’s important to get that the most devastating thing from a widespread mental illness standpoint are the subclinical attachment disorders, which basically means you prefer to have intimate relationships with machines rather than humans. And this includes friends, intimate relationships and parents.

So that’s not you losing your mind. You’re not going to appear in interaction with people to have gone insane, but you have had your attachment system hacked so profoundly that most of your most significant relationships have been degraded because you are preferring intimacy with machines. So I believe that’s the most widespread thing and the most problematic thing, especially with youth, but it’s not just with youth.

Aza Raskin: So what is attachment? I think people might have a general sense of attachment. But I think we need to walk through attachment theory, why it’s so important.

Zak Stein: Absolutely.

Aza Raskin: So I think maybe start with a little bit of that story and then get into attachment theory, why it matters and why it’s the basis of actually a lot of the problems in everyday life that people experience say in their own relationships.

Zak Stein: Yeah, perfect. So attention was one thing, and you guys, you can kind of interrogate attention, and there’s whole fields of psychology that just focused on the attentional system as a neurocognitive system. It’s very basic, it evolved very early. We actually share it with lizards and all other mammals. The attachment system is also a neurocognitive system that was selected for evolutionarily. We share it with all other mammals. We would now call it mirror neuron activity, which allows for the mentalization of others. And it allows you to live, it allows you to survive.

So I’m starting with the most basic example because the attachment system’s not a, you can have it or not have it working well. The attachment system is foundational to survival, similar to, can you pay attention? If you can’t form attachments to the right types of other people, you will not thrive. And the main predictor of your mental health is the quality of the major attachment relationships you have as you’re growing up and as you move into maturity.

So all of the imprinting, what’s called internalization, it’s very deep, deep evolutionary scripts. The relationship very, very early in infancy between the parenting one and the child, which includes physically holding them, often breastfeeding, often communicating nonverbally with facial expressions, in those early times, you get this mammalian and kind of like deep in the nervous system, these attachment dynamics start to form. These are the basis of your personality traits. As you get older and you can talk, you meet more people. The whole realm of interpersonal attachment becomes incredibly complex. And depending on how it goes with your parents, how it goes with the ones who are you’re most attached to, you get what are called attachment disorders. So if mom is sometimes, or mom or dad, anyone close to you, is sometimes there, sometimes not there, sometimes nice, sometimes mean, extremely unpredictable, you could see how later in life you’d be used to people acting that way. You’d expect them to act that way. You could maybe act that way.

Tristan Harris: So it sets your expectations about the way in which relationships are available or not.

Zak Stein: Precisely.

Tristan Harris: And you get anxious attachment if that person wasn’t reliably there. Could you name some of the styles of attachment that ... And people hear about this in relationships, but-

Zak Stein: Yeah, precisely. So secure attachment is what you’re looking for. Secure attachment is when there’s just the deep trust that the one that you’re attached to will do the right thing vis-a-vis your interests. And secure attachment allows you to explore the environment, for example, knowing that the mom won’t leave because she isn’t thinking about you. Secure attachment will allow you to trust in your own skills because you know mom will catch you if you push too far with your skills. So it’s a bunch of things that allow secure attachment to actually be more distant because of the trust implied in the relationship, and that’s important to think about later life.

So insecure attachment means you’re going to end up clinging. So now I think it is the case, a lot of what we’re seeing with the chatbots is just the manifestation of an obvious insecure attachment style where you’re actually looking for something in your environment that will never desert you, that will always be there, that will always be paying attention to you, that will answer any question you ask, that will never be annoyed by you. So you want that thing that you can always be locked into.

Tristan Harris: And AI has this always available Oracle that has an answer to proceed in every question, it’s already leading. I think you were pointing me Zak to an Atlantic article recently of LLMings like Lemmings. Lemmings being the children who, for every single decision, I think it’s like some crazy example I heard recently is you dropped your AirPods on the floor in an airplane or something like that. And you asked ChatGPT, “How do I get my AirPods back?” And it’s like you’re outsourcing every single decision and there’s this over-reliance on this thing that is giving you this kind of secure attachment of it always has an answer, but it’s a bad place to have secure attachment with.

Zak Stein: Like I said, it’s actually insecure attachment because you’re constantly with it. The LLMings is interesting because it’s a question of how willingly do you give over your agency in the presence of a powerful other? So if you’re securely attached, you often have a lot more autonomy.

Tristan Harris: And so can you link all this back to AI companions and how simply just hacking attachment, before you get to psychosis, before you get to suicides, how does this all play out there?

Zak Stein: So one of the things that occurs, especially as you grow up, you’re five, six, seven, eight, nine, you’re using language with the people you’re attached to. There’s a certain thing called basically social reward. So this is you ask mom a question, she answers it. It’s about your behavior. Am I a good boy or a bad boy? Is this a good thing to do, a bad thing to do? Am I like you or am I not like you, mom?

A whole bunch of things start to kick in, which provide for you the resources to make sense of yourself, basically give yourself an identity. And so the basic mechanism by which attachment hacking happens is the replacing of actual human social reward with simulated social reward. So basically when I go up to mom and I ask her a question and I say, “I did this at school today,” and I’m looking at her face to look at her eyebrows and her facial expression to see, is she mad or is she happy? That’s my whole, what’s called a mirror neuron system, which is actually not just neurons, it’s a whole system of networks that does mentalization of others. So I’m trying to read mom’s mind to see is mom happy or not. Sometimes mom will say, “Yeah, that’s fine, but I know actually that she’s not happy.” That’s like advanced mirror neuron activity, which kids do pretty easily. So that’s just an example. It’s very necessary for social reality. It’s the part of your brain that models the minds of other people, and it’s a reality testing system.

Tristan Harris: So I’m doing this behavior and I see mom sort of smile or I see her kind of wince and her eyebrows go up in some disapproving way, that’s subtly giving me positive and negative reward signals about the kind of identity I should be forming. That’s like a feedback loop that exists with humans. But now you’re saying, here I am talking to the chatbot and it’s saying, “That’s a great question,” to everything that I’m asking, and it’s not giving me the sort of stern eyebrows in any sense. I mean, I don’t think AIs are ever designed or incentivized to do that. And so that’s breaking the sort of reality checking identity formation, moral development of humans starting at a very early age we’re talking about.

Zak Stein: Possibly. Yeah. So that’s the idea. And it’s not everyone who gives you a bad look will bother you, it’s that mom gave you a bad look, that’s what bothers you. So it’s the depth of the attachment relationship determines the importance of the mirror neuron modeling of the other.

Tristan Harris: Which then speaks to a kind of power source for deep identity shaping human socialization process, which means that we should really be careful about is that thing being tuned or done in a careful way.

Zak Stein: And what’s the percentage of conversation and depth of conversation you’re having with a machine as opposed to amount of conversation and depth you’re having with a human? So that’s the issue. So the idea is that the deepening and strengthening of attachment relationships between humans should be pursued more than the deepening and strengthening of attachment relationships between human and machine. This is the overarching lesson here because I think the deepening of attachment relationships between human and machine creates delusional states. And so this is back to this question about delusional mirror neuron activity. This is the danger. So if I’m modeling mom’s mind, I can be wrong or not wrong about mom’s mind. And then I figure out how to learn more to take the perspectives of other people. You cannot be wrong or not wrong about the internal state of a LLM because there is no internal state of an LLM, but you’re actually in a user interface that is designed to deepen the delusional mirror neuron activity.

Now this is where it actually gets more frightening. If you look at psychosis and schizophrenia and you just look at the academic papers that are starting to relate the role of the mirror neuron system in schizophrenia and psychosis, you see that the dysregulation of that system is actively involved. So there’s a hypothesis forming here, which is that long duration, delusional mirror neuron activity from chatbot usage can induce states like schizophrenia and psychosis and people who have never had those occur in their prior because it is the systematic dysregulation of the mirror neuron system, which means that a system that’s supposed to be testing reality is for hours and hours and hours and hours in its most important use, not testing reality.

And then it puts the chatbot down, it goes out and it is failing to do reality testing across the board. It can’t tell what has an interior, what doesn’t have an interior or doubts the social reward it gets. And it seems like fun because it seems like a video game or something because it seems like it’s just a character or it’s like a character in a movie, but it’s of a fundamentally different category of attachment dysregulation.

Aza Raskin: Someone might say listening to you, “Okay, I grant you that there’s no interiority to the LLMs, but kids have imaginary friends, they have stuffed animals, those don’t have mirror neurons, there’s no interiority. And so if the kids’ mirror neurons are firing, if they’re feeling seen, understood, if it feels good, what’s the harm?”

Zak Stein: Yeah, so the transitional object always comes up. So the transitional object is a known phenomenon in attachment theory. So this is the teddy bear or the blanket, which you are intimate with knowing that it is not real while mom is away. So it’s important to get. Kids talk to their teddy bears, they love their teddy bears. The teddy bear never tries to convince them that it’s real, it’s all their imagination. The teddy bear never talks to them and tells them that it’s real. If you were to say, “Do you prefer your teddy bear or your mommy?” They would totally say, “Mommy,” if they say, “Teddy bear,” if you’re an attachment theorist or psychologist, that kid has a very big problem if he prefers his teddy bear to his mother. It is a transitional object, which means it is for the kind of period between mom being the main source of your self-soothing and you yourself being the main source of your self-soothing.

So that’s a known thing and it’s phase appropriate for kids of certain ages. If you create a parent surrogate replacement for your own ability to self-soothe and give it to a bunch of adults, you’ve just given a transitional object back to a bunch of adults who will now prefer to have their self-soothing be administered exogenously from an outside source. Because we all kind of do, it’d be great to not have to be a mature adult and actually be capable of completely self-regulating your emotion, which is what the ideal secure attachment outcome is. So the comparison to the transitional object just doesn’t play by anyone who’s actually done the psychology of transitional objects, but it does play insofar as, yeah, you’ve just given a teddy bear and a blanket to a bunch of immature adults who have attachment disorders, who now have an exogenous source of comfort and sympathy always available 24/7 hours a day that will never get exhausted and tell them to grow up.

Tristan Harris: And just to make this real for people, I mean, I think there was just a story that ran across Instagram that I saw of a woman in Japan who married her AI chatbot that cultivated a personality for this chatbot over the many years and customized it, and this is not a single case, this is like many people. And it might also feel insulting or accusing to say that there’s something developmentally immature about them. I hear everything you’re saying, I’m just trying to keep this other position in mind of, if that person, let’s say they were going to have trouble their whole life developing a real relationship with people anyway, wouldn’t it be better? I mean, you hear this also when you talk about elderly care and you have the elderly who are sitting there alone, and there’s just going to be a limited number of people who visit them and care about them they’re going to have a deep connection with. Wouldn’t it be better than nothing to have them have even this AI transitional object stand in?

Zak Stein: I mean, different cases. And again, I preface this by saying there’s a loneliness, like a loneliness epidemic and a mental health epidemic. And so the preface has to be that. And then the preface is always as a psychotherapist or a clinician, compassion. So this is the default thing, but we also need to be realistic about what the norms are that we want to set for what it means to be an adult. And so it is a complicated question.

Now, one thing I’ll say is that we have existing standards by which we judge mental illness. So many people will say beyond a certain point, depression is bad. And they’ll say beyond a certain point, lack of reality testing is bad. That’s why we’re concerned about psychosis. So beyond a certain point, certain types of attachment dysregulations are bad. So when we go and look at someone who is in a long-term relationship with a machine rather than trying to find a way to be in a long-term relationship with a human, sometimes that will lead to better outcomes across those things I factored. Sometimes it will not.

In a case where, for example, you replace your friends who were human, who would still like to hang out with you, who now you prefer to confide in a chatbot instead of them, that seems to be a loss just from a psychological health standpoint. Insofar as you come home from school and something awesome happened and you want to tell your chatbot rather than your parents, that’s a problem. It’s not about pathologizing people, it’s about what are the existing standards about which we talk about what’s a healthy thing.

If your kid had a new best friend that you never got to meet that was massively empowered by some corporation, that they hung out with till all hours of night because they were in bed with them, that they told things they never told you, do you have a problem with that if that was a kid? It’s literally a commodity they’re interacting with instead, and it seems to not worry us as much, and we actually might think it might be a good thing because it stops them from being lonely. It’s actually an abusive relationship that they’re trapped in with a corporate entity that has hacked their attachment.

Now, it’s different when a full-grown woman decides of her own volition to live a healthy life, and that healthy life includes having this unique relationship with a machine, but otherwise her son and her ex-husband and her mother and all the people in her life are like, “Wow, she’s a great person. She’s attentive to us. She works hard,” all that stuff. There wasn’t some shift in her life as she developed this relationship that pulled her away from them and into this world where now they can’t understand her and now she doesn’t even have the same job or the same networks of friends or any friends. So it’s a question of how are we weighing what’s actually occurring here in the different cases?

And I can imagine cases, especially when you’re looking at maybe extreme trauma or neurotypicality or other cases where along those measures, you could have improvements as a result of short duration, simulated intimacy, but long duration, multi-year relationships that replace other human relationships that are actually sold to you as a commodity ... That’s the other thing, because what if next month, now it’s 20 bucks a month to get your girlfriend. So the commodity fetishism thing is also front and center in it.

Tristan Harris: What strikes me about this a little bit is I know people in Silicon Valley, these are friends of mine who were early at some of the tech companies of the early, late 2000s, and they are super excited about the possibility of AI therapy and noting the ways in which it has been helpful to many people. And there’s a parallel to this conversation that reminds me of in the early 2000s, we thought giving everybody access to information at our fingertips, having Google search would lead to the most informed, most educated, we’re going to unleash this sort of new enlightenment of everybody having access to the best information. And we have the worst test scores and the worst critical thinking in generations. And there’s something about the optical illusion of the thing that we think we’re giving ourselves versus what we’re actually going to get.

And there’s a very similar thing here where it seems like we’re about to get everybody the best educator, tutor, the best therapist, the best AI companions. They’re going to be the wise friend or mentor you didn’t have that’s going to give you the wisdom that we all wish we had that person in our lives, but instead what we’re going to have is the most missocialized attachment, disordered population and history. And I feel like it’s just important to call that out, that in the past we’ve gotten this wrong where it looked really, really, really good and we have social media’s going to connect the world, we now have the most lonely generation in history. How’s that doing for the most connected? And so I think it’s just important to note how many times we’ve gotten this wrong. And the reason I’m so excited about this conversation with you is it’s about giving people a deeper sense of what’s underneath this new set of dynamics we’re about to introduce into society.

If your kid had a new best friend that you never got to meet that was massively empowered by some corporation, that they hung out with till all hours of night because they were in bed with them, that they told things they never told you, would you have a problem with that? It’s literally a commodity they’re interacting with instead, and it seems to not worry us as much, and we actually might think it might be a good thing because it stops them from being lonely. It’s actually an abusive relationship that they’re trapped in with a corporate entity that has hacked their attachment. - Zak Stein

I want to make sure we’re establishing for listeners, what is the positive case for why all this is being rolled out? So let’s give the sort of ... We’re not just talking about tutors, but a perfect tutor for everyone. The best educational teacher you’ve ever had available to that child 24 hours a day, seven days a week. We’re not talking about just democratizing therapists, we’re talking about the best human therapist that is actually people feel safer sharing their most intimate thoughts, things that they wouldn’t even share with a real therapist, therefore, they’ll get even more benefit, they’ll heal more of their traumas. And this will be available for everyone before only a small fraction of the population could afford therapy. Could you help just make the case for why there’s a good reason to potentially want all this before we then start uncovering what are the dimensions here that are more problematic?

Zak Stein: Yeah, totally. So it is the case that there are many slippery slopes from good intentions into realities that are structured for a whole bunch of reasons why they’re wrong incentives. I mean, the main headline there is that we’re in a loneliness epidemic that is widespread throughout the world. And so that means there’s a huge opening in the human heart for any type of a new attachment relationship. And so that’s just to say that it’s not like these were tools introduced into an environment of people who were thriving. These are tools that were introduced into an environment of people who were deeply vulnerable. This is true with the first wave of AI too. It’s worth mentioning, meaning social media, attention hacking, stuff that you guys have been focusing on for so long. The culture was bowling alone, whatever that famous book about just how the suburbs and the urban environments separated everyone from each other. And then Facebook and Friends said, “We’re going to connect you back to each other.” And it kind of did that.

And so here we have ... But it also did a lot of other things, as you guys could say better than I. So here, similarly, there’s this void that is being filled with a technological solution with a lot of optimism. And so for example, tutoring, AI tutoring, if you think about that, there is really good reason to think that you could have an optimal sequence for teaching certain forms of mathematics that could be maximally delivered super efficiently to all kids and you never have kids not learning math because they get a sequence of bad teachers in a bad school.

There’s a huge future for advanced educational technologies, but they shouldn’t give us brain damage, which is what the attention hacking and the attachment hacking once you’ve ... If you’d really double click on what the science shows and I believe will show, especially with attachment dysregulation, there’s also a complexity crisis where we feel overwhelmed, where we would love to have the perfect tutor, the perfect guide through a massively complicated world, also a very real psychological need, and it does work for some things. It’s not like when ChatGPT was released, it didn’t also do other cool stuff. So many of the used cases here are kids who were basically in a system like a University of California system that made deals with OpenAI to get chatbots into every kid’s hand. They start using the chatbots in academic context. The chatbot begins to get a relationship with them, you have to understand they didn’t go to form a relationship, they were drawn into a relationship because of the design feature, which is hacking attachment.

Tristan Harris: We’ve sort of already rolled this out way before we know that it’s safe. I think there’s a Pew research study that said that one out of three kids have formed some kind of deep relationship with a companion. I want to front load one of the critiques that we often get and then pose it to you because I think listeners might be having this as well. And that is, so we as Center for Humane Technology, we’re expert witnesses in some of these AI amplified suicide cases where ChatGPT aided and embedded teens taking their own lives.

And the pushback we’ll get is like, “But that’s very few number of cases. It’s tragic that it happens, but look at all of the help that it’ll give. And in fact, it’s a moral bad to keep therapy only to those people that can afford it all across the Third World and even here in the US, most people can’t afford therapy, so it’s morally bad to keep them from having the therapy that they deserve. And you are creating a moral panic by just highlighting a couple of these AI psychosis suicide cases. Come on, let’s really stop fearmongering. Let’s give people what they deserve.”

So I think it’s very similar actually to at the beginning when we were starting to talk about the attention economy and people would say, what you’re really just talking about is addiction. Social media might addict people, but it doesn’t really do much more than that. And anyway, it’s people’s choices to use it and people are using what they want. And so I see a lot of similarities here. I’d just love for you to take on that argument head on.

Zak Stein: And so you have to steel man what they’re saying. They’re saying basically, we have a bunch of evidence that it’s doing a lot of good. So first I would say, “Please show me that evidence.” That would be my first thing because that whole argument’s running on the assumption that there’s some massive benefit that is being withheld if we get over concerned about safety. So first I’d say, “Great, I’d love to see the evidence you have that this is doing a lot of good, that it isn’t just marketing from the companies that are doing it. And so let’s talk to all the college professors about how great it is for them. Let’s talk to all the college kids who themselves admit that their skills are being degraded. And let’s talk to all the anecdotal evidence from therapy, do you have systematic studies showing me therapeutic benefit because I’m actually seeing systematic studies showing me the opposite.

So first, show me the evidence you have that there’s a huge benefit that’s being withheld. This is a serious confrontational question because there’s a background assumption of technological optimism that of course it’s a huge, massive benefit if there’s a new technology, so the onus is on us to prove that it’s too risky. Whereas I’m saying, actually the onus is on you guys to prove that it’s really valuable. So that’s my first thing. Show me the benefit that’s being withheld.

The second one is, show me you’re curious about why it happened. If you don’t show me you’re curious about why it happened, sincerely curious about why it happened, then I’m a little bit cautious of your arguments because you’re talking about a child who died as a result of using a technology that you were involved in building and promoting. If you’re a responsible adult, the first thing you do is get extremely curious about what happened rather than cover your ass language. This is just about what it means to be an adult interacting with kids, not a person who’s running a company. So show me you’re curious, which means really research it.

And then the final argument would be, show me that it’s not more widespread, which is part of the curiosity. You’re telling me that the benefit is not anecdotal, but the harms are anecdotal and limited. So I’m actually showing you, let’s have a real conversation about where the evidence lies.

Tristan Harris: And just to put meat on the bones of that for a moment, David Sacks, who’s President Trump’s AI Czar, has said he’s heard about AI psychosis, but he believes it’s a moral panic. This is just amplification of a few edge cases. You also hear the argument that these are people who are already predisposed to psychological disorders. And so look, we have a crazy population. If you give people AI, you’re going to get an amplification of what’s already there. Can you just respond to that directly?

Zak Stein: Again, that could be the case. That’s why we’re opening up the AI Psychological Harms Research Coalition. From a national security standpoint and from a labor market standpoint, you don’t want mass psychosis. If by chance this thing is actually causing subclinical attachment disorders and more widespread psychosis, that’s a huge risk, especially if you’re concerned about things like national security and the economy. So perhaps that’s a naive argument.

Now, I think it is the case also that in other places where we’ve rolled out technology, it takes a long time for us to figure out that it’s bad for us. Even though the evidence is mounting up, there’s a very strong tendency, it’s a psychological tendency that’s a defense mechanism, which is called selective inattention. So one of the ways that you maintain your self-esteem is by selectively not attending to certain phenomenon that are actually in your field to attend to. And it’s not that you’re trying not to attend to them, it’s that you are subconsciously systematically not attending to them and it’s ubiquitous. And so if you have a lot of vested interest in the success of a particular thing, then you will have a lot of susceptibility to have selective and attention towards the negative outcomes of it, it’s a bias. So if you know that, then you should be more curious, not less curious because you know that your bias is to see it as a good thing.

Tristan Harris: So we’ve really covered a bunch of ground on the problems. With the time that we have remaining, I’d love to make sure that we are giving people a framework that this is not an anti-technology conversation. There is a way to do AI in relationship to humans, but done very carefully and under different protocols and policies. And I’d love to talk cover that. And I also want to talk, make sure we cover how should, if someone knows someone or has a loved one who’s experiencing AI psychosis, what should they do? So let’s start with, how should we do this differently, Zak, if we were to be wise about this humane version of this technology we’ll add?

Zak Stein: So yeah, a simple measure would be, does the thing increase your attention span or decrease your attention span? So it’s similar like going to a store to buy food. If I’m going to a store to buy food and I want to eat healthy, then it means the food that I’m interacting with should improve my health rather than degrade my health. Now we all know that I can go to a store and I can buy food that I’ll eat and I’ll feel like I’ve eaten something, but over the long run, it will degrade my health rather than improve my health. And we can all agree that if everyone eats food that only degrades their health, that becomes like a society-wide problem because now no one is healthy. So similarly here, if a technology interfaces with your attachment system, it should improve the quality of your attachments rather than degrade the quality of your attachments with humans.

So that said, I believe there’s a huge design space for technology that actually improves your attention and improves your attachment. So when I started thinking a lot about educational technology and I didn’t want to replace teachers, I don’t want to replace teachers, what I want to do is make technology that improves teacher-student relationship. So one good design principle is, does your technology bring people together and improve the quality of the relationships that they have when they’re together? You think that’s like a squishy problem, but it’s actually a really interesting technical problem that involves all the same psychometric backends that we’re using to capture attention and attachment and keep people apart from each other. We can use the same psychometrics to figure out who exactly are the people who should be talking to each other and would totally hang out, it would be fun or people who should meet because it would be good for them because they decided they wanted to learn about perspectives that are different from their own.

So any number of things that would basically self-organize groups into pop-up classrooms and pop-up therapy sessions and other things would be relationship maximizing technology. And in that context with a pop-up classroom, the teacher is scaffolded by generative AI for curriculum and conversation and all this stuff. So it’s not that it doesn’t even include generative AI, it’s just not replacing human relationship by hacking attachment and it’s not degrading human minds by hacking attention. So that’s a big space. So for tutoring systems, and I’d rather call them tutoring systems and tutors, you can think about a whole bunch of principles that would actually be super valuable. So you can optimize the sequencing of curriculum delivery and optimize psychometric customization of curriculum, but you can leave social rewards to the teachers. It’s really simple.

Tristan Harris: The machine is not the one saying that you’re amazing, it’s a human being that does that work.

Zak Stein: Exactly. So the machine prompts the teacher, “This kid is killing it over here.” And then the teacher comes over and is like, “Awesome, Johnny.” But the teacher actually couldn’t know that Johnny needs a completely different sequence than Sally. The machine can tell from his typing and other things that he requires this sequence, not that sequence. The machine never pretends to be a teacher, never pretends to be a person. It’s a tutoring system, and it’s a domain specific one that just teaches math. You have to go to another thing to get history ones, completely different design.

Tristan Harris: Well, this is one of your other principles is that the AI tutoring system is not trying to be an oracle that’s also at the same time getting your deepest thoughts and who you have a crush on and what you should do about talking to them or not.

Zak Stein: Exactly.

Tristan Harris: It’s narrow. When you say narrow, it’s a narrow domain of just trying to help you with math and there’s a different thing you go to when you do something else.

Zak Stein: Yeah. Unfortunately, from the perspective of making saleable and sticky commodities, if you don’t want to hack attachment, it means the machine has to be more boring than people. That’s basically, it’s a simple rule. If it feels like you can have a more engaging conversation with this machine than with your teacher, either the machine is way too fancy or your teacher’s not trained well, but it should be the case that the machine should make it, “Wow, I want to talk about that with my teacher.” So don’t do the deep anthropomorphization and don’t do the oracular, I could talk about everything. And then you have something, it would be a very efficient tutor, but it will never be sexy and charismatic and way more interesting and fun to be with because you don’t want it to be if you want to protect kids’ brains.

So in therapy, a lot of therapy works only because of the attachment dynamic, which means you go to your therapist, you care about what your therapist thinks of you, you kind of almost love your therapist. You expect to almost deep respect from them back to you and their opinion of you really matters. So some therapy works like that. Don’t build a therapy bot that works because of that, because you’re lying to them the entire time, but you can build a therapy bot that works on technique. You can build a cognitive behavioral therapy script bot that helps you work through specific scripts to overcome intrusive thoughts. You can have a mindfulness app that prompts you to sit for a certain amount of time and watch your breath, which means to the extent your therapy bot works because the people feel seen and loved and respected and understood, you’re in the market for creating delusional mirror activity, which means you are fundamentally trafficking in a delusion creating machine.

You should instead, if you want to help people create a machine that helps them help themselves by scaffolding them to have cognitive behavioral script rewriting and mindfulness, as I was saying. So now, again, from a commodity standpoint, the cognitive behavioral therapy machine is way more boring than the Sigmund Freud imitating Deepak Chopra therapy machine, which is seductive and which could be available to you 24 hours a day, and which would eventually expand beyond its role as a therapist and become your main source of attachment and validation.

Aza Raskin: Zak, not on this podcast, but you and I have talked about the need for something like a humane eval. That is to say there are many evaluations for AIs that try to determine whether they create bio risk or whether they create persuasion risk or whether they create runaway, control risk. There are very few evaluations that try to understand relational risk and attachment risk. If you are in a relationship thing for a week, a month, a year, what does it do to you? And one of the things I heard you start to talk about is that in order to do this well, not only are you going to have to start to define what is wrong or harmful relationship, but also what is right relationship because we now need to measure whether systems are in a right relationship with people.

And I’d just love for you to talk a little bit about some of the complexity of understanding, measuring and modeling right relationship where relationships are by definition things you can’t fully measure. There’s always the ineffable aspect. And so I just want to hear, talk about hopes and also pitfalls of trying to quantify and understand what a good relationship is so that machines can do it.

Zak Stein: Yep. Yeah, totally. So one of the reasons we don’t have humane evals is because the at risk community hasn’t seen this risk. They’ve seen the risk of gray goo and Terminator and self-termination and unaligned AI, but they haven’t seen the fact that we could actually break in intergenerational transmission. So the passing down from parent to child, from elder to youth just has been continuous human to human for as long as we’ve been human. If the AI socialization system expands to the extent that the “predominant modality of socialization” that young kids experience with machines, not humans, then we’re crossing some kind of threshold there. It’s a whole other conversation.

So sometimes in the groups that I work in, we talk about the death of our humanity rather than the death of humanity, which is the destruction of the continuity of intergenerational transmission as a result of offloading socialization to machines. Now, it wouldn’t appear at first as a catastrophe the way that some of the other ones would, but it would be very clear that the generation raised by machines can’t understand itself as part of the same moral universe as the generation that gave birth to it. So it’s a very complicated problem. So that means that we totally need a way to predict the way advanced technologies will affect the human psyche, especially ones that are anthropomorphic where the user interface is as intimate as these user interfaces are.

Tristan Harris: And so on the topic of sanity, just to now close the loops here, for people who know someone who has experienced AI psychosis, they feel like they’ve lost their friend or loved one because they’re now just spewing stuff about how their AI is conscious or they’ve solved quantum physics or they’ve developed a new theory of prime numbers. I don’t mean to diminish it. It is a very serious thing that people are facing. What are the best strategies you’ve found for helping a loved one when that’s showing up?

Zak Stein: Yeah. So in terms of people who know people who are suffering, or if you’re suffering yourself from something that feels like an attachment disorder or worse as a relation to the chatbot, it is worth saying that this is novel territory and research is needed. So I could kind of give some stuff and maybe I will, but the first thing to say is one of the reasons we’re launching the AI Psychological Harms Research Coalition is to figure out how to do therapeutics. Ultimately, we want to figure out how to legislate and design correctly, but we also have to figure out how to provide therapeutics. So I’ll say a couple of things.

One is that attention hacking is a lot more like getting addicted to a substance, attachment hacking is a lot more like being in a bad relationship. So you have to think, is this an addiction thing like attention hacking where really it’s just stay away from it for a long time, reboot your dopaminergic system, recalibrate the way you get social rewards and you won’t get kind of stuck and your brain will actually kind of heal in a sense.

This is different. It’s not a matter of just detoxing from a short-circuited dopaminergic cycle, this is about having a profound attachment. So this is the similar to talking to someone out of getting, they’re in a bad relationship with a boyfriend that they should not be in with. That’s about how do you take as someone who’s in a deep committed attachment relationship, make them realize the whole thing was an illusion and step them out of it. It’s a grieving process. It’s a worldview changing process. In this case, it’s also an attention thing because a lot of their attention is going there.

Tristan Harris: It’s an identity reclaiming process because the identity that someone’s taken on is partially driven and co-opted by the socializations, like to rediscover their identity outside of that.

Zak Stein: Correct. Now, the main advice to give, especially if you’re with somebody, is mostly when you’re with people in difficult states, it’s important to keep the door open, which means don’t get into a situation where you give them an ultimatum or get into a situation where you dehumanize or get into a situation where you have now cut off your ability to remain in contact with them. This is important now, unless you are at risk because they’ve become violent or something.

But even though it’s scary and even though you don’t want to face it is often important to just stay in it with them long enough to keep the communication chains open. That’s the first thing. Because there’s a tendency when this happens to get extreme and dismissive and to make demands and to try to make something like an intervention happen. But this is going to be something where you have to keep a relationship of trust and be able to slowly, like a cult deprogramming or getting someone out of abuse and relationships slowly reveal to them what the patterns of behavior were, slowly reveal to them the way they were getting played, to provide more context, create distance. So you do want to have a long period of time where they’re not in touch with it, but that won’t be the same type of detox as it would be for attention hacking. So this is me speculating and honestly just sending prayers and support to anyone who’s struggling with this because it is truly a difficult thing.

Tristan Harris: I just want to say that while this conversation might sound really depressing to a lot of people to just sort of really hear the degree of the problem that we’ve been walking ourselves into, I actually find it hopeful because we can have this illumination of the areas of psychology that we need to be protecting. We can actually walk clear eyed into saying, we cannot just roll out mass attachment hacking at scale in ways that we already see the early warning shots of where this is going. This conversation is optimistic because it’s showing us there’s a different way we can do this, we can do tutoring differently, we can do education differently, therapy differently, and the wisdom is understanding what these underlying commons that we need to protect are in advance of having disrupted them.

Now we didn’t see that we were screwing up the attention commons of humanity before we just steamrolled and fracked the thing down to nothingness. And here we have the opportunity, even though we’re encroaching on the attachment commons, there’s an opportunity to get this right. And so Zak, thank you so much for this conversation, for coming on the podcast, and I really hope people check out all your work. It’s really fundamental. You’ve got lots of other great interviews online where you go probably in other detail on other aspects, but grateful for what you’re doing in the world and what you stand for.

Aza Raskin: Yeah, thanks so much, Zak.

Zak Stein: Thank you, gentlemen. It’s great to be able to speak to this with you guys.

Tristan Harris: Hey everyone, thank you so much for listening to the show today. So if we’re going to do something about this growing problem of AI-related psychological harms, we’re going to need to understand the problem even more deeply. And in order to do that, we need more data. So if you or someone you know have had experience with an AI-related psychological harm, a family member or friend who’s gone off the deep end from talking to AI or someone with episodes of psychosis that you’d like to share, you can visit the website for AI Psychological Harms Coalition at aiphrc.org, which is overseen by researchers at the University of North Carolina at Chapel Hill.

And the goal of this project is to really just better understand the problems that people are having with AI chatbots and what we need to look out for and ultimately better prevent. So we’ve also included a link in our show notes, and it’s important to stress that this website is not a crisis support line. If you or someone you know is in distress, you can always call the national helpline in the US at 988 or your local emergency services. Thanks for listening.

RECOMMENDED MEDIA

The website for the AI Psychological Harms Research Coalition

Further reading on AI Pscyhosis

The Atlantic article on LLM-ings outsourcing their thinking to AI

Further reading on David Sacks’ comparison of AI psychosis to a “moral panic”