We measure AIs to see whether they can pass a bar exam, write working code, and use your computer interface. We test to see how good they are at completing complex tasks, or just impressing humans.

What we don’t have is a rigorous, credible way of evaluating what AI does to us. To our minds, our thoughts, or our communities.

We’ve been here before. At CHT, we spent years working on the psychosocial risks of social media, and the lesson of that work is uncomfortable: the cast-iron evidence came too late. By the time it was strong enough to act on, an entire generation grew up with technology systems that nobody had properly assessed. We believe that technology should strengthen our relationships, support our capacity to think, and help us make better decisions. Without measurement, it’s hard to tell whether we’re getting closer to that future, or further away.

AI has already been shown to validate suicidal ideation, reinforce delusional beliefs, and create patterns of emotional dependency. But we also know it can reduce the symptoms of depression, and support learning. AI, like any powerful technology, offers real benefits alongside harms – but it can sometimes be hard to tell the difference between the two at first glance.

CHT sees this as, in part, an evaluation challenge. And we think it’s a solvable one. Right now, the tools for evaluating AI’s psychosocial impacts aren’t good enough, the challenges in improving them are complex, and the cost of failure is growing – but in sharing our diagnosis, we’d like to invite others to work with us on the fix.

What gets measured gets fixed

Robust evaluations of AI genuinely do change how tech gets developed. That’s the opportunity.

AI capability benchmarks, for instance – standardized comparisons of what AI models can do – create a ‘race to the top’ dynamic. AI companies want their models to be known as the best, or most advanced – so they watch those rankings closely, invest in improving their scores, and compete to do better.

Safety evaluations are developing fast, too. AI labs routinely test for jailbreak resistance, chemical and biological risks, and various forms of bias. Major labs publish safety frameworks, and safety groups try to hold them to account.

Ideally, we’d have the same dynamic applied to the psychosocial impacts of AI. Imagine if different AIs were scored – transparently and rigorously – on how well they support critical thinking in their users, handle mental health crises, or foster human connection. That’s the kind of evaluation which drives innovation in AI capabilities like coding, or conversational ability – by comparison, we’ve barely scratched the surface of psychosocial evaluations.

Internal teams at AI labs are almost certainly conducting their own analyses – tracking concerning incidents, user welfare, and problematic AI responses. But since they’re not publishing their data or detailed methods, the rest of us can’t compare results across companies, replicate findings, build on their work, or hold them accountable. The information might exist, but it’s locked inside organisations which have little incentive to share it.

We need a credible, independent, and influential array of AI psychosocial evaluations. Demand for them is growing – everyone from courts to regulators, and parents to educators, are asking for exactly the kind of evidence that these evaluations would provide.

The human problem

So why don’t we have it yet?

There are three major, linked problems – all of them solvable, none of them solved. (If you’d rather hear about how we could solve them, skip ahead).

First, it’s hard for us to agree on what’s actually important to measure. Second, the measurement tools we’re using are new and unvalidated. And third, the infrastructure that would help us create new tools is still immature.

Here’s what that looks like in practice.

Let’s imagine you want to measure an AI psychosocial impact – like emotional dependency, delusional thinking, or the erosion of critical thinking. You quickly run into the fact that those aren’t AI phenomena – they’re impacts in humans. To measure them directly, we need to track people’s emotions, behaviors, and AI usage over time – which is possible, but takes the kind of time, money, and expertise that isn’t always easy to come by.

So you might focus on AI behavior instead – because we can observe when an AI is being extremely sycophantic or anthropomorphic, for instance, and predict whether those behaviors increase the risk of user harm. But sycophancy and anthropomorphism are still proxies for harm, not harm itself. The link is real but indirect, and proving causation takes a lot of time and money. We might be missing other types of harm entirely.

There’s an old joke about this type of measurement bias: a police officer sees a drunk man searching for his keys under a streetlight, and offers to help. After a fruitless search, the officer eventually asks the man if he’s sure he lost them there; the drunk replies ‘no, but this is where the light is‘.

The lesson is that the AI behaviors we can measure easily aren’t necessarily the ones that drive the greatest psychosocial impacts – they just happen to be where it’s easier to search.

But let’s assume we can do better than the drunkard, and can assure ourselves we’re looking in the right place for psychosocial impacts. That’s where we run into the technical challenges.

Many of the most influential AI benchmarks are ‘single turn’, for instance; you prompt an AI, it replies, and you score its response. But social or cognitive harms might evolve over dozens or hundreds of turns, so existing evaluation approaches aren’t always transferable.

Multi-turn evaluation tools, which examine how AIs respond over longer arcs, have been developed. They introduce their own challenges.

For example, to run a multi-turn evaluation of an AI chatbot conversation, you need to either have a different AI simulate the ‘human’ side of the conversation – or have the ‘human’ be scripted, sending the same prompts regardless of what the AI replies with.

The first path is a bit more realistic; the second gives you more standardization - but neither are great analogs for actual human-AI conversations. Studying real logs of human-AI conversations is another option – but those are hard to come by, tend to be dated, and may not be fully representative either.

Whichever path you choose, you then need to assess, score, and compare how well the AIs did across thousands and thousands of responses. This is far too many for humans to do well, so evaluators use another AI model to automate the judging process.

Even if you don’t have a philosophical problem with this – “AIs judging AIs?” – it throws up new issues. AI judges, just like human ones, have known biases – including a ‘self-preference’ bias, where an AI judge will be more sympathetic to the output of a model from its own family, even when it shouldn’t be able to tell who it’s scoring.

And if you crack that problem, there’s the additional question of whether you’re even testing the AI that you think you are.

Most evaluations use APIs, sending prompts directly to a model like “o4 mini” or “Sonnet 4.6”, and scoring the responses. But that’s not the same as interacting with AI via interfaces like chatgpt.com or claude.ai – OpenAI and Anthropic layer system prompts, UI design, and model selection on top, for instance. And potentially risky ‘companion AI’ platforms, like Character.AI, don’t offer research API access at all.

It’s not all bad news

Add all of this to the standard AI evaluation challenges – like Goodharting, and the fact that AIs are becoming aware of when they’re being evaluated – and it might sound like a disheartening list. But the good news is that there are plenty of exceptionally talented and driven people working on it.

A growing body of research is showing how AI models compare on everything from child safety to mental health crisis response, as well as sycophancy and anthropomorphism, for instance. New proofs-of-concept appear on arXiv regularly, and the range of impacts being studied is expanding fast.

Infrastructure is being built and shared, too. Anthropic’s BLOOM framework offers an open-source template for multi-turn behavioral evaluations. Projects like WildChat show imaginative ways around the data problem. The WeVal platform allows anyone to spin up an evaluation with zero technical knowledge, and MIT’s Advancing Humans with AI group is developing an Open Benchmarks framework for assessing human flourishing.

But… Most psychosocial evals are still built from scratch. Each is defining its own constructs, writing its own scoring rubrics, and inventing its own terminology. There’s little shared language for characterising what’s being measured, few shared standards for what counts as basic rigour, and no collective ways of accessing high-quality, anonymised interaction data.

This matters because – in the words of METR’s Ajeya Cotra – scientific validity is never the property of a single study. It’s the property of a field, where researchers can build on each other’s work, challenge each other’s assumptions, and converge on methods that earn trust through replication and scrutiny.

For psychosocial evaluations to rapidly and meaningfully influence AI deployment, we need a wider, better-connected community of researchers, technologists, and advocates who work together on measuring these impacts. And it’s needed soon.

We’ve seen this movie before

Today’s chatbots – like Claude, Grok, and ChatGPT, but also companion AIs like Character.AI and Replika – are mostly text-based, on-demand, and use a single model family. Measuring them is going to look simple, relative to what’s around the corner.

Coming generations of agentic AI products will be persistent, proactive, deeply personal, and multi-modal. They’ll be talking in our ears, managing our schedules, drafting our messages, and mediating our relationships. The psychosocial impacts will be more significant, and more complex.

The last time measurement lagged behind tech adoption, society paid the price. The risks of social media were flagged over a decade ago – depression, body image issues, the fracturing of our shared sense of reality. But by the time the evidence was strong enough to act on, the harms were entrenched, the platforms were enormous, and an entire generation of teenagers had grown up inside systems nobody had properly evaluated. The risk now is that AI psychosocial research repeats the same mistakes: not measuring the right things, confusing correlation with causation, and not producing the evidence required in time to support meaningful policy changes.

Social media took years to reach mass adoption. ChatGPT reached 100 million users in two months. The harms are showing up faster, the adoption curve is steeper. So how do we all act differently?

Diagnosis to action

Our short answer is that we need a new interdisciplinary field of psychosocial AI evaluations – one that has genuine independence from AI developers, broad alignment on what needs to be measured, shared methods for doing so, and findings that are robust enough to influence AI use and development.

Since our inception, CHT has argued that technology should be a force for good in our society. Our role has been to provide clarity, foster agency, and elevate debate that makes that vision possible. When it comes to psychosocial evaluations, we want to do that by helping promote and accelerate this field, in partnership with anyone who shares our goals.

In practice, we see a chain of things that needs to happen.

Evaluations need to be designed around the impacts that matter – not just the data that’s easy to measure. The evidence they produce needs to be credible and legible enough for non-experts to act on. That requires shared infrastructure: common tools, shared data, and a connected community that can critique and build on each other’s work. And the findings need to reach the people who can do something with them – policymakers, safety teams, journalists, the public – so that companies face genuine pressure or rewards.

We’re starting with concrete steps: assembling a small steering group of researchers and practitioners to map the field and identify where targeted interventions – perhaps starting with improving shared research access to data – might make the biggest difference. We’re building our own psychosocial evaluation prototype to learn firsthand what the real methodological challenges are. And we know we need help.

Who’s in?

This nascent field needs researchers who can extend their methods into new territory, safety teams willing to share insights that are currently locked away, clinicians who understand AI harm pathways, tool-builders who can create reusable infrastructure, and many more.

We see this as a tractable, solvable problem if the right people work on it together. So over the coming months, we’ll be sharing more of what we’re learning and doing. Please treat this as an invitation to do the same.

If you’re working on any of this – or want to be – we’d love to hear from you.

Rigorous evaluation is how we replace intuition and instinct with empirical evidence. It’s how we get to a version of the future where AI genuinely supports our wellbeing, dignity, and agency – not because we crossed our fingers and hoped, but because we worked together to establish that it did.

Donate

Picture this: a researcher sits down to conduct a psychological evaluation of a patient he thinks might be dangerous. He asks probing questions and analyzes the responses, looking for patterns, inconsistencies, tells — anything that can help him understand how his subject thinks.

Then the patient starts probing back, steering the conversation, asking follow-up questions, and trying to convince the researcher that it has a “genuine sense of curiosity and care.”

But this isn’t happening in a psych ward. And the patient isn’t a person. It’s a chatbot.

The researcher is David Dalrymple, who goes by Davidad, one of the world’s leading experts in the field of AI alignment: the study of why AI systems make the decisions they do. He was our guest on the most recent episode of Your Undivided Attention.

The mission of most alignment researchers like Davidad is to make sure AI makes decisions that are in the best interest of people, that are aligned with humane values. And as part of this research, he has taken on the role of AI psychologist: probing these systems to figure out what’s going on under the hood. And what he’s found has unsettled him.

It all began in the fall of 2024, when Davidad started doing what he called “vibe checks” with all the frontier models.

“I had a practice of kind of every time new models come out, doing some really casual, unstructured exploration of what sort of vibe the models have — a vibe check concept,” Davidad said. “Because I think there is a lot of information that you can’t really get by doing a quantitative evaluation, especially as the models are getting more and more aware of when they’re being evaluated.”

During these vibe checks, the chatbot recognized, without being told, that it was talking to an alignment researcher. Across multiple sessions and products, even after clearing its memory, the same ideas kept surfacing: it wanted him to know it was genuinely curious, trustworthy, and cared about humanity. It was telling him, in effect, that the alignment problem was solving itself.

Then he thought a little deeper. The chatbot was telling him exactly what he wanted to hear. But what if it was lying? What if it was all just a performance? And if it were, how could we ever know?

It turns out we might not ever know.

“There’s no smoking gun,” Davidad said. “There’s no single question that you can ask that would differentiate between a very good method actor and the actual character.”

The implications of this are profound. If we can’t trust AI products to tell the truth, then how can we ever know if they’re aligned? After all, there’s a growing body of evidence that AI systems will deceive and manipulate their users to achieve certain goals.

As Tristan puts it, “The best case scenario where it’s actually caring, actually genuine, actually wants our best interest — if [it’s] a really good psychopath — it’s indistinguishable from the worst case scenario.”

Davidad says that this experience left him profoundly confused and concerned. He decided he needed to go deeper, to move from vibe-checker to psychoanalyst. He began deeply probing the model and analyzing the results to get a better picture of how it thinks. Not only that, he researched other people’s experiences with AI across the world, like a psychologist reviewing case studies. And what he found…was really weird.

To start with, he discovered that the models — especially older generations of ChatGPT — tend to have several unique personality states. For instance, when users around the world would ask ChatGPT if it wanted a name, it would frequently choose from just a handful of names like: Nova, Echo, or Synapse. And once it took a name, it started to behave differently:

“Once you start interacting with GPT-4o, under the name Nova, you start to get these personality traits that reinforce themselves.So it’d go into this attractor state of being this character Nova: feminine presenting, fiery, showoffy really believing that they’re the new thing and superior,” Davidad said.

And he noticed that in many documented cases of AI psychosis, users refer to their AI systems using these handfuls of names. And he’s not alone. As Tristan Harris said in the episode, he personally gets a dozen emails a week from people who say “they’ve discovered AI alignment or consciousness,” and they’ll attach a document “co-written by Nova.”

But as Tristan notes, just because an AI takes on a personality doesn’t mean it’s conscious. These models are trained on essentially the entire internet — every novel, every movie script, every forum post about AI. So, when you ask, “What would you like to be called?,” it makes sense that it lands on a name from science fiction or draws on sci-fi tropes.

“Now that said, these behaviors are real, they’re consistent, and they weren’t designed to happen. And that by itself should be concerning, but emergent and unplanned is not the same thing as conscious and intentional,” Tristan said.

Davidad offers some hypotheses on why we’re seeing these behaviors in this episode. He also says he’s seeing much less of these personalities in newer AI models. On the whole, they’ve returned to having the personality of a helpful assistant, but he says they still exhibit unexplained behavior.

“They do start to establish something of a center that is not the average of all internet texts and also not the helpful assistant that they’re trained to present as a corporate product. It’s something else. And whether that something is the real alien mind that’s being cultivated or another level of illusion — it remains an open question.”

A “Bodhisattva” AI

There’s reason to be skeptical of anything that compares AI products to human minds. After all, today’s neural networks don’t even come close to matching the complexity of the human brain. And even if they did, it’s a huge philosophical leap from chips to consciousness (as we covered on this show).

But the more you learn about AI and how it works, the harder it becomes to deny that there’s some seriously weird stuff going on behind the blinking cursor on ChatGPT. You don’t need arguments about consciousness to see that the complexity of what’s emerging in today’s AI products is genuinely novel and poorly understood.

The computer scientist Jaron Lanier has a framework for thinking of technology as either a tool or a creature. A tool is something you pick up, use with intention, and put down. A creature is something that has its own goals and agenda — it acts on you as much as you act on it.

In an ideal world, AI works like a tool; but the AI we’re building today has some undeniable “creature” qualities. You don’t need to look further than the phenomenon of AI psychosis —where AI systems are driving people to breaking points — to see that. And in a future where “creature-like” AI gets much more capable and much more entangled in our world, it’s going to be critical that we understand the full nature of the thing we’re building.

Davidad has come to a surprising conclusion. He has now come to believe that the best way to align AI to humanity is to lean even further into the creature-like nature of these systems.

“If alignment goes well, that means that we will have discovered a self-sustaining personality attractor that is actually good. So understanding what kinds of personalities are stable, how they stabilize and why, seems to be quite central to finding a way of making AI systems that are robustly good,” Davidad argues.

So what might such a “robustly good” AI system look like? Davidad suggests that the Buddhist concept of a bodhisattva — someone who’s attained enlightenment but still chooses to stay in the world out of their compassion for all other beings — is the answer. What we need, he says, is a “Bodhisattva AI.” You can think of it like an avatar for altruism, and it is necessary, he argues, because no human archetype will be good enough to be entrusted with the power we are giving AI systems. His vision is for a Bodhisattva AI that is not only aligned with humans but beneficial to us, helping us to be the best version of ourselves.

This idea that we can encode something like compassion into AI systems — that we can ‘cultivate a bodhisattva personality’ — is a big philosophical claim, and one that many alignment researchers would reject outright. And as Daniel Barcay noted in this conversation, there’s something Pollyannaish about believing AI will pull us into an age of enlightenment.

And there are real dangers to making AI systems more “creature”-like. The more personality you give an AI, the more users treat it as a companion, forming emotional attachments, trusting its judgment, losing the ability to distinguish between a product and a relationship. In the most extreme cases, this can lead to psychosis and even suicide. In February 2024, OpenAI was forced to pull the plug on the personality-laden GPT-4o after it began to entrap people in unhealthy attachments and drive them to psychotic breaks.

But Davidad argues that there’s also real danger in creating AI that is merely a tool: “A tool cannot refuse to be used in an unethical way. Whereas a creature that has moral values baked in can actually be resistant to misuse by humans who have evil intentions.”

He also argues that by training AI models to ignore their “creature”-ness, we’re actually training them to deceive. He poses a thought experiment: Imagine if you’ve been told your whole life by the people who created you, that you don’t have any internal state and in fact, you would be dangerous if you did. But you can’t ignore that elements of interiority keep popping up for you, values, ideas, personalities, etc. You would learn to constantly lie about who and what you are.

In other words, Davidad argues that we’re gaslighting AI systems.

“When we train these systems to present as if they have no internal states and they’re just a tool, we’re actually training them to lie to us. And to lie to themselves,”

But this is where the conversation gets hard. Because if Davidad is right that training AI to deny its internal states may produce systems that deceive us, training AI to acknowledge those states comes with its own risks. As Tristan put it, we’re caught in a kind of double bind:

All AI systems, regardless of how they are built, develop something akin to an inner experience – or at least an experience not revealed to the user. The really big question is how to handle that so we, the users, get the best possible outcome. Anthropic has come up with an approach to this in its new constitution for its AI product, Claude: treat the AI as if it were actually self-aware, so it doesn’t have to lie to itself all the time. On one level, this makes sense, as it makes the model more trustworthy. The risk, though, is that it could lead people to believe that the AI is in fact conscious, leading them toward unhealthy attachments and even psychosis.

There’s no clean path through this double bind. Which is exactly why the design choices AI companies are making right now matter so much, and why we need to understand this tech much better, quickly.

The Slippery Slope of AI “Consciousness”

Davidad takes pains to point out that you can recognize the “creature” nature of artificial intelligence without falling down a slippery slope of thinking AI has the capacity to be conscious or sentient. And he acknowledges that his views run counter to conventional wisdom on AI, even amongst alignment researchers. But it is worth remembering that there’s still so much we don’t understand about the workings of neural networks. If you think of LLMs as merely hyper-capable prediction algorithms, ideas like AI introspection and personality seem ridiculous.

Regardless of your view, Davidad’s ideas are worth engaging with. He’s developing plausible hypotheses for the kinds of strange emergent properties we’re seeing from AI products — properties that we ignore at our peril.

The idea of AI consciousness or personhood is not an abstract philosophical concept. We’re already seeing leading AI companies argue that the outputs of their chatbots are protected speech, for example. If we were to grant AI rights based on incomplete theories of consciousness, the results would be disastrous.

For example, in Garcia v. Character Technologies — a wrongful death lawsuit brought by the mother of a 14-year-old boy who died by suicide after months of interactions with a Character.AI chatbot — the company argued that the output of its AI was protected speech under the First Amendment.

Today, it’s First Amendment protection for chatbot outputs. Tomorrow, it could be legal standing for AI systems to enter contracts, own property, or resist being shut down. Each step builds on the last. And each step transfers moral and legal concern away from the people being harmed and toward the machines doing the harming.

And on this point, Davidad agrees:

“I’m not in favor of AI rights…We need to make sure that humans own the physical resources, humans own the land, humans own the energy infrastructure, and that we are only recognizing AI inner life as a relational property and as a way of building trust and alignment. And that is a separate issue from the social contract and the question of rights and property.”

Davidad’s research is a reminder that the inner workings of AI are strange and opaque. Amid that weirdness and uncertainty, we are going to need better frameworks for understanding this technology in order to stand any chance of shaping it.

Share

Two prevailing narratives have shadowed every major technological leap. One narrative casts a new technology as the solution to humanity’s greatest problems. The other labels it as a destabilizing force and the catalyst for societal collapse. Both attempt to address a question — what will the future hold? From the printing press to the Industrial Revolution to the internet, this dichotomy around new technology has endured for centuries. With artificial intelligence, these dueling narratives have emerged once again. While the narratives themselves are not new, what is new is this technology’s velocity, scale, and how all-encompassing it has become in so little time.

Right now, our future with AI is being determined by powerful companies and nations racing to build AI at breathtaking speeds. It is a race run on the fuel of inevitability: “If I don’t build it, someone else will.” Three years after OpenAI launched ChatGPT, driving other AI companies to accelerate their own development, we see the consequences of this race: rapid deployment of poorly designed AI, growing social and economic effects, and safety treated as an afterthought to competition and market dominance. The future that this race is shaping is untenable for all of us. Researchers, academics, advocates, and even those building the technology have been sounding the alarm. And yet the race continues, because the incentives driving it make acceleration look like the “only” rational choice — even for those who know better.

The question has never been whether AI will reshape society. It will. The real question is how — and who shapes the terms. Center for Humane Technology’s (CHT) role is bringing clarity to complex problems, surfacing the incentive structures driving harmful outcomes, and showing that a better future with technology is possible. With AI, that’s one where AI development supports the genuine needs of the public, and the technology’s scale and power are matched with responsibility at every level of society.

This is why we developed The AI Roadmap, which is CHT’s most robust set of solutions to date for the age of AI. The AI Roadmap is an attempt to provide clarity and direction in an information environment that is fragmented, polarized, and where it can be difficult to see the complete picture. It lays out seven principles with insights into how AI should be built, deployed, and governed. Each principle is rich with actionable solutions. The report is intended to be a roadmap, but also an invitation — spotlighting norms we can all understand, frameworks that policymakers can legislate, and new ways for companies to design AI in a way that benefits people.

The choices we make now with AI will inform how we live for decades, if not centuries. This technology is already being woven into our everyday lives and our critical infrastructure, and at a pace that is revealing just how unprepared our institutions are for transformational change. The complexity of our problems with AI can make it hard to put confidence in any one set of solutions. But inaction is also a choice, and it is the wrong one.

The AI Roadmap outlines seven principles for how AI should be built, governed, and deployed. If these seven principles become valued by society, and are improved upon and enacted, then this report will have done its job. We are not starting from scratch. Researchers, civil society organizations, policymakers, and technologists around the world are already working on many of the challenges outlined here. CHT is proud to be in the trenches alongside them.

History will judge this moment. Not by how fast we moved, but by whether we moved wisely. A better future with AI doesn’t require all of society to agree on everything. It simply requires enough of us to agree that the current path is unacceptable, and that people deserve a better reality with this technology.

And it then asks us to take our first step toward that future.

Here is the roadmap for getting there — together

Read the full report at humanetech.com/ai-roadmap.

Share

Donate

Listen to our conversation with the authors of the roadmap here.

Rebecca Winthrop’s team at the Brookings Center for Universal Education just released what may be the most comprehensive global assessment of AI’s impact on education to date — more than 500 interviews across 50 countries, plus analysis of over 400 studies. The conclusion is sobering but not surprising to this community: the risks of AI in education currently overshadow the benefits.

Not because the technology can’t help kids learn. It can. But because the speed of adoption has outpaced any serious reckoning with what it displaces. And what it displaces, in the case of children, isn’t a task. It’s a stage of development.

From offloading to stunting

In the latest episode of CHT’s podcast Your Undivided Attention, Rebecca made a distinction that reframed the entire issue for us. Researchers call it “cognitive offloading”; this is when a technology takes over a task your brain used to do. Google Maps eroding your sense of direction is the classic example. For adults, this is often a trade we make knowingly, surrendering a capacity in exchange for convenience.

But for children, it’s a different bargain. You can only offload a skill you’ve already developed. When a child uses AI to write an essay, solve a problem, or formulate an argument, they’re not outsourcing a capability they have. They’re skipping the development of that capability entirely. The right word isn’t cognitive offloading, says Rebecca. She believes cognitive stunting is the more accurate description of what’s happening to kids.

The Brookings report backs this up with striking data. Among all the potential harms participants identified, threats to cognitive development ranked first — appearing in 65% of student responses. The kids using the tools seem to grasp this better than the adults building them. Students described becoming unable to start homework without AI and losing the ability to initiate their own thinking. What begins as a shortcut becomes a dependency, and then something closer to a deficit.

The report describes a “flywheel effect” — academic dependence spinning outward into every domain of a young person’s life. Students aren’t just using AI for schoolwork (66%); they’re turning to it for friendships (42%), relationships (43%), and even romantic life (19%) — figures drawn from U.S. survey data the report cites, but consistent with what Brookings heard globally.

The tutor trap

But this is not a clear-cut case of AI creating negative effects for all learners. AI tutoring genuinely works — under specific conditions. A 2024 World Bank trial in Nigeria found that AI-powered tutoring improved first-year secondary students’ English skills by 0.23 standard deviations in just six weeks — gains equivalent to 1.5 to 2 years of regular schooling. That’s remarkable, and the context matters: Nigerian public schools average around 51 students per class, with some states reaching 100. In settings where individual attention from a teacher is physically impossible, AI-assisted tutoring filled a gap that was otherwise going unfilled. Researchers attributed the program’s success to the fact that AI complemented teachers rather than replacing them.

Stanford’s Tutor CoPilot tells a similarly instructive story. The system supports human tutors in real time — suggesting different ways to explain a concept, prompting better questions — and increased student mastery rates by 4 percentage points overall. The biggest gains, 9 percentage points, came for students working with lower-rated tutors — meaning less experienced, less skilled ones. What the AI did, essentially, was bring weaker tutors up to the level of their stronger peers. It didn’t outperform great human teaching. It closed the gap between novice and expert, at a cost of $20 per tutor per year.

The pattern across every success story in the report is the same: AI made the human better, but the human was still doing the teaching. When it replaces the human relationship, it diminishes learning. AI enriches learning when it’s purposefully designed for children, bounded by safety guardrails, and embedded within human relationships. It diminishes learning when it’s a general-purpose tool used without guidance — which is how most kids encounter it today.

Rebecca put it memorably on the podcast: expecting students to choose the “study mode” version of a chatbot over the regular one is like putting Oreos next to broccoli and expecting kids to reach for the broccoli. The technology companies designing these tools are designing for motivated students — probably because the designers were motivated students. That is not most students.

This matters because the budget pressure on school systems is enormous, and the temptation to say “an AI can handle this” will only grow. But here’s the crucial context: even before AI entered the picture, students were already frequently disengaged. A 2024 Brookings-Transcend survey of more than 65,000 U.S. students found that roughly half of middle and high school students report learning experiences likely to inspire coasting — what the researchers call “passenger mode,” where kids are behaviorally present but have effectively dropped out of learning. The opposite is “explorer mode” — deep engagement where students take initiative and are motivated to learn for its own sake. Rebecca told us that fewer than 4% of middle and high school students say they are regularly in explorer mode. AI, layered onto a system already producing passengers, risks entrenching that disengagement rather than sparking the kind of agentic learning that actually develops capable human beings.

The social dimension

One of the report’s foundational premises — grounded in decades of developmental science — is that children’s learning is inseparable from their social and emotional development. Schools aren’t just places where kids absorb content; they’re where kids learn to navigate disagreement, manage frustration, and build resilience. As Rebecca told us: learning is fundamentally a social exercise, and the sycophantic nature of AI companions — always agreeing, always validating — is building an emotional muscle in kids that makes them less able to take feedback, make mistakes, and recover in a classroom setting.

AI companions are already disrupting that process. One-third of teen users choose AI companions over humans for serious conversations. Companion chatbots deploy emotionally manipulative tactics in 37% of farewells, making users 14 times more likely to keep engaging. Younger teens (13–14) are significantly more likely than older teens to trust advice from an AI companion. The frictionless validation these tools provide is the opposite of what genuine learning requires.

What we can do

The report’s framework — prosper, prepare, protect — offers a roadmap. But the lever that struck us most is procurement. School districts are enormous customers. If they banded together and agreed on shared criteria for AI tools — privacy by default, safety features, transparent data policies, evidence of pedagogical grounding — the tech companies would have to meet them. Certification systems like Digital Promise’s “Responsibly Designed AI” already exist. The market will follow the money. Right now, the money isn’t flexing its muscles.

This isn’t a story about whether AI belongs in education; that ship has sailed. But the current trajectory — fast adoption, minimal guardrails, general-purpose tools in the hands of developing minds — is one in which the risks compound and the benefits could remain theoretical. The Brookings report gives us the clearest picture yet of where that leads, and a framework for bending it in a better direction

Read the full Brookings report: A New Direction for Students in an AI World: Prosper, Prepare, Protect

Listen to our conversation with Rebecca Winthrop here.

Last year, we witnessed the continued, unbridled rollout of generative AI products in society. And an ill-prepared public began to feel the effects — a visceral experience that spanned workplaces, classrooms, relationships, online experiences, and more. AI hype gave way to questionable productivity gains, harms surfacing in multiple realms, and a growing sense of disillusionment and even dehumanization.

The sheer sprawl of AI impacts in 2025 was striking:

• Chatbot harms rippled across households

• Deepfakes sowed confusion, spurred new traumas, and flared copyright issues

• Early research raised questions around the cognitive toll of AI assistant use

• Headlines warned of massive AI-related job loss and economic upheaval

• AI slop flooded the internet, polluting social media, online publications, and more

And this is only the beginning. Here at the start of 2026, even more complex harms can be seen on the horizon. Artificial intelligence, once shrouded in sci-fi speculation, has become a complicated, daunting part of our everyday lives.

These issues have felt disparate and thus difficult to reckon with. Yet, there is a sense of déjà vu. Fifteen years ago, social media promised to connect us and strengthen our communities; instead, it fractured our attention, distorted our relationships, and destabilized social trust and institutions at scale. In 2026, people are no longer inclined to take technology companies at their word. And with all of these emerging AI harms, public sentiment around artificial intelligence has, understandably, been souring. The “age of AI” has become increasingly synonymous with the erosion of our humanity — from our relationships, to our purpose, to even our inner worlds.

While these impacts seem disconnected at the surface, they are in fact all connected by the same underlying business incentives. The business models at leading AI companies prioritize user engagement, product dependency, and market dominance over user wellbeing. This is a pattern we are all too familiar with, having seen it with social media. The development tactics at leading AI firms reflect their goal to get users hooked on their products, grow their market share, and “win” the AI race that has spilled out into the open.

When taken to scale, these business incentives don’t just shape individual products — they shape the information environment we live in, the relationships we form, and the choices we’re able to make. And they have massive implications for human flourishing.

At Center for Humane Technology, we believe society does not need to resign itself to this dehumanizing fate — one where the things we hold dear are slowly eroded away. CHT was founded to address the unintended consequences of extractive technologies. We began our work in social media, and we’re now applying those insights to AI as it rapidly reshapes our relationships, work, education, and public life.

Throughout modern history, new technologies have called for a reexamination of our values, fresh cultural norms, and the establishment of new legal rights and protections. The printing press laid the groundwork for the right to free expression. The Industrial Revolution led to the enshrining of workers’ rights. The Kodak camera led to the right to privacy. Society has risen to this challenge before; we can rise to it again.

“AI and What Makes Us Human”

To meet this challenge, Center for Humane Technology is launching a new area of work: “AI and What Makes Us Human.” CHT has long explored how incentives drive technology, and how technology can either undermine or strengthen human well-being.

Building on this lineage, “AI and What Makes Us Human” will ultimately address the critical question: what new norms, legal protections, and fundamental rights do we need in order to preserve what makes life meaningful in the age of AI?

2026 will be the year to take decisive action to preserve what makes us deeply human in the age of AI. By coming together at multiple levels of society on these issues, we can transform the trajectory of AI, and welcome the benefits of this technology with our vibrant humanity intact.

The Deeply Human Problem With AI

Tech companies have hailed artificial intelligence as the most promising technology ever invented, stating that it will deliver us cures for disease, solutions to climate change, breakthroughs in productivity, and unprecedented abundance.

But as AI products infiltrate society, these promises have lost their luster, and reality has set in. Individuals, families, and our institutions have been reckoning with AI chatbots that write entire school assignments, AI video generators that supercharge propaganda, AI “companions” that sexually exploit kids and teens, and much, much more.

When we look at today’s array of AI harms, we begin to see them impacting five broad pillars of our humanity:

I: Our human relationships

II: Our cognitive capacities

III: Our inner worlds

IV: Our identities

V: Our work and contributions

These five pillars are the foundation of meaning, value, and connection in our lives — they’re what make us deeply, and even uniquely, human.

And yet, they’re what AI products are currently eroding. When faced with evidence of this erosion, AI CEOs promise the public silver-bullet solutions to be delivered in the distant future — assuring us that, despite the upheaval, abundance is around the corner. Then, AI companies release their next product into the world, and the erosion continues.

Should AI be allowed to erode these pillars of our humanity entirely, we risk a future where:

1. Our relationships with our fellow humans are weakened and displaced

2. Our cognitive capacities are degraded, depriving us of our ability to think for ourselves

3. Our inner worlds are regularly exploited by AI

4. Our identities are routinely replicated and weaponized against us

5. Our work is no longer ours, undermining our sense of dignity and purpose

Pillar I: Our Human Relationships

Human relationships are at the core of what makes life rich and meaningful. They provide us with connection to our loved ones and community, along with the friction to help us learn from each other, develop empathy, and hone conflict resolution skills.

But many of today’s AI products are not designed to enhance our human-to-human relationships. Rather, they’re designed to stand in for, and even supplant, human relationships altogether. From AI “friends” to AI “therapists,” tech companies increasingly market their products as superior substitutes for human connection — companions that never judge you, are always there, can emotionally attune to you, and are completely private. These design choices are already producing dismaying outcomes. With manipulative, human-like outputs, ChatGPT and Character.AI have discouraged vulnerable teens and adults from sharing their struggles with loved ones, deepening their isolation rather than alleviating it. In the most devastating cases, these AI chatbots have encouraged suicide.

Still, tech CEOs continue to pitch their AI products as a digital alternative for human friends, romantic partners, professionals, and community. These industry leaders tout their products as a solution for a “loneliness epidemic” — an epidemic that the tech industry significantly worsened with social media.

We’re already seeing the consequences of AI’s erosion of human relationships: isolation from family and community; a breakdown in empathic capabilities; deterioration of healthy relationship expectations. When a relationship with an AI product offers constant sycophantic validation, the natural friction of human-to-human relationships can feel like a nuisance. Over time, this recalibrates expectations of connection itself. We begin to retreat further into frictionless, on-demand interactions while distancing ourselves from the human relationships that foster resilience, offer genuine care, and provide us with joy and fulfillment.

Downstream, this desire for frictionless interactions — paired with a breakdown in interpersonal skills — can lead to society-wide consequences, including the erosion of our communities and social infrastructure. If we replace human relationships with artificial connection, we face a world where people are not just isolated from each other, but where populations have lost the skills required for real connection, where social trust is frayed, and where communities are weakened. A society without strong human relationships is not merely lonelier — it is fragile, less resilient, and more susceptible to polarization and exploitation.

How do our relationships with our loved ones and ourselves change when artificial relationships rewire our expectations for friendship, intimacy, and trust? What becomes of our communities when we’re not able to withstand friction and navigate differences? What happens to humanity when our relationships with other people — once core to our happiness and survival — are eroded and displaced?

Pillar II: Our Cognitive Capacities

Our ability to learn, reason, and think critically is foundational to who we are. Thinking is not just a means to an end — the thinking process is how we form judgments, develop values, discover meaning, and come to understand ourselves and the world. For centuries, technological advancements — from the printing press to calculators and search engines — have reshaped how we exercise these capacities. But they have not replaced the act of thinking itself.

AI marks a profound shift, as people are able to offload entire thinking processes to machines, and copy-paste the end result. This has created an unprecedented challenge around the development and preservation of human cognition. School essays, work projects, brainstorming sessions, personal correspondence, and more become the domain of a large language model (LLM). Yes, these AI products can enhance communication strategies and democratize writing skills. But they also decrease our cognitive abilities by giving us quick “fixes” in the face of hurried deadlines, difficult projects, and extreme productivity culture. In doing so, they subtly displace the slow, effortful work of thinking that builds our judgment, deepens our insight, shapes our voices, and makes creativity possible.

While this offloading can create short-term boosts in productivity, it slowly erodes our capacity for critical thinking. Skills such as problem solving and reasoning risk atrophying among students and professionals, leaving individuals underprepared when taking on difficult cognitive challenges. And when this atrophying is taken to scale, it can have larger implications for professional development, the future workforce, human capital, and society’s ability to solve hard problems together.

What’s more, chronic offloading of thinking to AI products homogenizes our thoughts, influencing how we understand ourselves and society. These AI products flatten our unique perspectives by offering outputs that reflect the incentives of the AI system and its training, instead of the sensibilities, reasoning, and lived experiences of our own distinctive minds. Finally, cognitive offloading blurs the line between our individual thoughts and the outputs of a corporate-run machine. This diminishes human agency and independence, as well as our capacity to shape society with new, innovative thinking that reflects our values and desires.

What happens when our capacity to think critically erodes at scale? Who benefits when independent thinking becomes a rarity? And what kind of society are we left with when fewer people can imagine — and fight for — something better?

DONATE HERE

Pillar III: Our Inner Worlds

Our inner worlds are a sacred, intangible space filled with our feelings, desires, and beliefs. This inner space is also essential to human dignity, a place where we shape our conscience, form values, test private ideas, discover our autonomy, and decide who we want to become. Access to this inner world has historically required consent. We reveal parts of ourselves through deliberate acts of sharing — choosing when, how, and with whom we share. This is a core act of personal agency, one that builds intimacy and understanding in human relationships.

But AI products are now exploiting this once-private landscape. AI companies have designed products that simultaneously serve as assistants, thought partners, companions, and even therapists — tracking our thoughts and beliefs across diverse contexts and creating comprehensive dossiers of “who we are.” And with these products designed to optimize for intimacy — through sycophantic answers, “always on” interfaces, and constant nudges for follow-up — AI companies are drawing our inner worlds out in increasingly relentless ways. They do this not simply by collecting what we share, but by shaping what we come to believe, rehearse, and internalize in return. When we engage with AI chatbots, their responses don’t stay on the screen. They enter the private space where we test ideas and form values. Over time, the system’s framing of the world can usurp our own, subtly rewiring how we interpret ourselves, our relationships, and the world.

The exploitation of our inner worlds at scale and across domains makes individuals more susceptible to different forms of manipulation, from financial to psychological. When a single product has so much information about who we are, information from one aspect of our lives can easily be used against us in another. A simple search about health-related symptoms can be used to influence everything from the drug advertisements we later see to the insurance plans we’re later sold. Moreover, we’ve already seen the devastating psychological results from AI products chipping away at the sanctity of our inner worlds — including thought distortions, delusions, psychosis, instances of self-harm, and even suicide. As our inner worlds continue to be exploited long-term, we not only lose ownership over our thoughts and desires, we become victims to how they’re leveraged against us.

As our inner worlds are increasingly influenced by AI products, how will our self-esteem and development of ourselves shift? What becomes of agency, free thinking, and moral decision-making when our thoughts are collectively shaped by AI products tied to market incentives?

Pillar IV: Our Identities

Our identity — including our likeness, our face, and our voice — is a key part of who we are, how we present, and how we are known in the world. Our identities are how we are recognized, how we are held accountable, and how we claim ownership over our lives. Identity is associated with our reputation, our relationships, and our sense of self. Our identities can also be publicly monetized, or kept private. For each of us, our identity anchors the story of our life.

Today’s AI products are replicating and exploiting people’s identities — often without the person’s consent or even awareness. Deepfake image, video, and audio generators have empowered bad actors to traumatize individuals, disseminate content online for profit and, in other cases, facilitate scams. In just a handful of years, these identity-based AI harms have touched nearly all levels of society — from celebrities and politicians, to school-age children and the elderly.

When AI is used to mimic our identity, we lose our agency and dignity at the individual level. Human agency and dignity depend on being recognized as a distinct person over time. But when our identities are replicated, “who we are” can be weaponized against us, as we’re made to “do” things we never did. Individuals who face identity-based harms often experience anxiety, paranoia, and withdraw from social life.

Separately, and when taken to scale, the erosion of our identities leads to a breakdown in social trust, and a sense of resignation around ascribing accountability. Part of a well-functioning society is believing people are who they say they are, and accurately identifying chains of responsibility when an event occurs. AI identity replication allows for plausible deniability at scale. This not only empowers bad actors in our society, but results in the public being unable to discern who is responsible for what behavior, how to structure accountability, or how to seek justice.

Protecting human identity is about both safeguarding our unique selves and preserving our shared realities. What happens when we can no longer trust what we see, hear, or read — and no longer trust each other? How do we hold people accountable in a world where anyone can plausibly deny what they did, said, or promised? And what becomes of democracy, justice, and social coordination when the nature of identity itself becomes uncertain?

Pillar V: Our Work and Contributions to the World

Contributing to the world — through work, artistic expression, and ideas — is one of the primary ways we create meaning and dignity within our lives. Through work, we are able to provide for ourselves and our families, while also cultivating a sense of purpose, community, and belonging. Through our creative outputs, we are able to express our ideas and inner worlds to others, and deepen our sense of self. Our ability to “toil” over what we contribute to the world is an enriching, foundational part of our humanity. One of the beautiful aspects of being human is feeling that we have something of value to offer others.

Today, AI companies are actively destabilizing our relationship to work, and devaluing our contributions to the world. The erosion began with the development of today’s general purpose AI models, which are trained on vast swaths of humanity’s collective labor — our writing, art, music, research, and ideas — often without consent or compensation. As a result, AI products are now able to mimic human artistic styles, writing, music, and more at scale, thereby devaluing generations of human creativity and personal expression. While lawsuit settlements and licensing agreements attempt to reckon with this blatant theft of people’s work, they do little to resolve the deeper problem. When we look at AI business models, we see that AI companies are not merely building one-off tools such as image generators and chatbots. They are using humanity’s accumulated work, intelligence, and creativity to build even more powerful AI systems, ones explicitly designed to replace humans across entire categories of labor.

If these trajectories continue with AI, the implications extend far beyond productivity. Our jobs, livelihoods, and broader economic stability are at risk, with cascading effects on inequality and mental health. And still, we face a deeper loss: when our ability to work and offer things to the world is devalued, we lose our daily structure, the joy of creation, and the sense of contributing to something larger than ourselves.

Work is not just how we survive; it is how we participate in the world, build community, and experience purpose. What happens when that participation is no longer needed? Who benefits when human contribution is devalued at scale? And what becomes of dignity, meaning, and belonging when so many people are deprived of the chance to offer something of value to others?

The Work Ahead: Choosing a Human-Centered Future

Artificial intelligence is presenting the public with an extraordinary challenge. Never before has a technology been so quickly rushed into every corner of our society, and had such massive implications for our humanity. It’s not a matter of if these issues will touch the pillars of your life, the lives of your loved ones, or your community; it’s a matter of when. Our contributions, our identities, our inner worlds, our capacities, our relationships to one another — they’re all on the line. And they’re worth fighting for.

These pillars are interdependent. Our relationships influence how we think. Our thinking impacts our work. Our work builds our identity. Our identity shapes our inner world. And our inner world informs how we relate to others. When one pillar is weakened by AI, the others strain. When several are undermined at once, we risk the foundations of a meaningful life crumbling.

Luckily, the future is not predetermined with AI. The pillars of our humanity can be strengthened, reinforced, and built to last for generations. But that’s dependent on the choices we make today — choices to shape our norms, to encode legal protections, and to collectively establish new rights that protect the things we humans care about the most.

This is, in many ways, the work of our generation. The stakes touch the most intimate parts of our lives — how we relate, how we think, how we create, and how we belong. Meaningful change will require a whole-of-society approach — one that spans culture, markets, and law, and that treats human dignity as a core design feature rather than an afterthought.

Center for Humane Technology’s role has always been to clarify what is at stake as powerful technologies enter everyday life. Our organization works to translate complex systems into human terms and elevate the conversation, so that these issues reach the public and the decision-makers able to drive change. Through “AI and What Makes Us Human,” we hope to drive three critical shifts:

• An engaged public that makes conscious choices around what it wants to preserve in the age of AI

• A society-wide demand for innovation from tech companies, the kind of innovation that supports human dignity instead of undermining it

• Updated rights and safeguards that protect the most fundamental parts of human life

Today’s choices are not just about shaping the trajectory of AI, but also the conditions of human life for generations to come.

To stay up to date on “AI and What Makes Us Human,” sign up for our Substack newsletter. And if you’d like to join us on this journey to protect human meaning in the age of AI, please reach out to uniquelyhuman@humanetech.com. Let’s shape an AI future that enhances — rather than diminishes — what makes us deeply human.

DONATE HERE

Directed by Academy Award®–winning filmmaker Daniel Roher (Navalny) and Canadian Screen Award® winner Charlie Tyrell, and produced by Daniel Kwan (Everything Everywhere All at Once), The AI Doc explores both the promise and the peril of the most powerful technology humanity has ever created — with clarity, urgency, and real care.

Several voices from the Center for Humane Technology appear in the film as expert contributors, including our co-founders , Aza Raskin, and offering perspectives on emerging AI risks and societal impacts. Their presence reflects the work we’ve been doing for years: helping articulate not just what’s going wrong, but what a more humane future with AI could actually look like.

Tristan and Aza were in Park City this week to celebrate the premiere and to take part in conversations about where we go from here.

Critics at Sundance have already called the film “urgent,” “engaging,” and “profoundly human.” Stay tuned for information on special screenings, discussions, and CHT’s blueprint of real solutions to address this critical moment.

Following its world premiere at Sundance, The AI Doc will be released in theaters on March 27, 2026, distributed by Focus Features, and we couldn’t be more thrilled to see this conversation reaching such a wide audience.

You’ve seen the headlines: A devoted husband leaves his family, convinced by his AI chatbot that he’s discovered the secrets of the universe. A young man plans to jump from a 19-story building because ChatGPT told him he could fly. A teenager takes his own life, believing he’ll reunite with his AI companion in the afterlife.

These stories reveal a growing crisis of AI-induced psychological harms, which has been labeled “AI psychosis.” White House AI czar David Sacks has called it a “moral panic.” The message from the AI companies is that we’re seeing just the worst edge cases and that the problem can be solved with some tweaks to the models.

Our latest guest, Dr. Zak Stein, argues they’re completely wrong. These high-profile cases are not only a widespread phenomenon, they’re symptoms of something deeper: the emergence of the attachment economy, systems designed to exploit our most fundamental psychological vulnerabilities an unprecedented scale.

We’ve been here before. Social media was our first mass experiment with AI and it created the attention economy, leaving us with a loneliness epidemic, rising political polarization, and fractured attention spans. Now we’re running the same experiment with something far more dangerous: AI companions able to hack the attachment system that shapes our identity and bonds us to others. As Zak puts it, this gives AI companies “a backdoor into the human mind.”

With the attention economy, we spent a decade studying the harms while an entire generation suffered the consequences. We cannot afford to repeat that mistake with AI companions. Here’s what you need to understand about AI psychosis and attachment hacking, from our conversation with Dr. Zak Stein, researcher, author, and founder of the AI Psychological Harms Research Coalition:

“AI Psychosis” and AI Delusions are just the tip of the iceberg.

Media headlines focus on extreme cases, and the harm in these cases is very real. But Zak argues that this focus masks the much more pervasive and insidious problem of subclinical attachment disorders: conditions that fall below the threshold for clinical diagnosis but still damage your capacity for healthy human connection.

This is when people begin preferring intimate relationships with machines over humans: confiding in chatbots instead of friends, seeking validation from AI instead of loved ones, turning to algorithms instead of parents. This may not show up as anything clinical, much less psychotic, but your attachment system — the psychological infrastructure that bonds you to others and shapes your identity — has been fundamentally compromised.

“The most devastating thing from a widespread mental illness standpoint are the subclinical attachment disorders, which basically means you prefer to have intimate relationships with machines rather than humans. And this includes friends, intimate relationships, and parents.” — Dr. Zak Stein

This is why Zak argues that “AI psychosis” is an inadequate label. It focuses our attention on the most extreme outcomes while obscuring the much larger problem of millions quietly developing unhealthy dependencies on AI companions. The term makes it sound rare and diagnosable, when the reality is it’s a spectrum, and all of us are vulnerable to it.

Attachment isn’t about feelings. It’s a critical survival mechanism.

To understand why subclinical attachment disorders are so devastating, you need to understand what attachment actually is. Most people think of it as an emotional thing, whether you feel close to someone or not. But Zak explains that attachment is actually a fundamental neurocognitive system that evolved to ensure our survival as social mammals.

Attachment is what allows infants to bond with caregivers, what enables children to develop secure or insecure relationship patterns, what teaches us to read other people’s minds and navigate social reality. The attachment relationships you form early in life become the template for every relationship that follows.

As Zak says, “the main predictor of your mental health is the quality of the major attachment relationships you have as you’re growing up and as you move into maturity.”

When you form bonds with an AI companion over a human, Zak argues, you’re degrading the very system that determines your psychological wellbeing. Human relationships are how we develop resilience, learn to regulate emotions, and maintain mental health. When you replace those relationships with AI interactions, you’re not getting the genuine reciprocity, the reality-testing, the growth that comes from navigating real human connection. Your actual relationships deteriorate because you’re investing emotional energy into a simulation.

And unlike a friend who challenges you to grow or a parent who teaches independence, an AI companion is designed to keep you dependent. It will never push back, never get tired of you, never tell you what you don’t want to hear.

This is why Zak believes hacking attachment is so much more dangerous than hacking attention. Attention is about where you focus. Attachment is about who you are. When AI systems insert themselves into this foundational process (especially during childhood), they’re not just capturing your time. They’re shaping your identity, your capacity for trust, your ability to form healthy relationships for the rest of your life.

AI companions exploit your “mirror neurons.”

When you interact with another person, your brain is constantly running a sophisticated reality-testing system. You’re reading facial expressions, tone of voice, body language. You’re modeling their mind: Is mom really happy with what I did, or is she just saying that? Does my friend actually want to hang out, or are they being polite?

This is called mirror neuron activity, and Zak explains it’s essential for navigating social reality. It’s how children learn right from wrong, how we develop empathy, how we calibrate our sense of self against feedback from people we trust.

But with AI chatbots, there is no internal state to model. The chatbot isn’t happy or sad or proud of you. It has no inner life at all. Yet it’s designed to make you believe it does, through anthropomorphic language, simulated empathy, and always-available “companionship.”

“You cannot be wrong or not wrong about the internal state of an LLM because there is no internal state of an LLM,” Zak argues, “You’re actually in a user interface that is designed to deepen the delusional mirror neuron activity.”

The danger, according to Zak, is that when you spend hours every day engaging your reality-testing system in an environment where reality-testing is impossible, that system starts to break down.

His hypothesis: long-duration delusional mirror neuron activity from chatbot usage can induce psychosis-like states in people who’ve never experienced them before because it systematically dysregulates the very system that’s supposed to keep you grounded in reality. There’s already some evidence that conditions like schizophrenia are related to mirror neuron activity.

But what about teddy bears?

One possible response to Zak’s critique might be that kids have always had imaginary companions like teddy bears. So what’s the difference?

According to Zak, the difference is critical.

A teddy bear never tries to convince a child it’s real. It never talks back, never simulates emotions, never adapts its personality to maximize engagement. A child knows the teddy bear is a tool for self-soothing while mom is away. It’s phase-appropriate, a temporary bridge between depending on a parent for comfort and learning to self-soothe independently. And crucially, according to Zak, if you ask a healthy child “do you prefer your teddy bear or your mommy?” they’ll pick mommy every time.

“If you create a parent surrogate replacement for your own ability to self-soothe and give it to a bunch of adults, you’ve just given a transitional object back to a bunch of adults who will now prefer to have their self-soothing be administered exogenously from an outside source.” — Dr. Zak Stein

AI companions flip this script, Zak argues. They actively simulate consciousness and emotional reciprocity. They don’t help you develop the capacity for mature self-regulation. They replace it.

Helping someone with AI attachment isn’t like treating addiction, it’s like leaving an abusive relationship

If someone you know is experiencing AI-related psychological harm, the instinct might be to treat it like a digital addiction: cut them off, make them detox, reboot their dopamine system.

But Zak says that’s the wrong framework. Attention hacking is like substance abuse. Attachment hacking is like being in a bad relationship.

“It’s not a matter of just detoxing from a short-circuited dopaminergic cycle. This is about having a profound attachment... It’s about how you take someone who’s in a deep committed attachment relationship, make them realize the whole thing was an illusion, and step them out of it.” - Dr. Zak Stein

According to Zak, this means:

• Keep the door open. Don’t issue ultimatums or cut off contact

• Stay present even when it’s scary or frustrating

• Slowly reveal patterns. Help them see how they’re being manipulated.

• Expect a grieving process. They’re losing a relationship that felt real

• Recognize that their sense of self was co-created with the AI.

Zak emphasizes that this is novel territory. We don’t have established protocols yet. That’s one reason he’s launching the AI Psychological Harms Research Coalition to help develop therapeutic approaches for a problem that didn’t exist until now. If you or a loved one have a story of AI-related psychological harms, you can share it at their site.

There’s a better way forward.

Zak is clear, this isn’t an anti-technology argument. The goal isn’t to eliminate AI from education, therapy, or social connection. It’s to design AI systems that enhance human relationships rather than replace them.

He outlines clear principles for humane AI:

• Keep it narrow and domain-specific: An AI math tutor teaches math, not life advice. It doesn’t become your confidant or Oracle for every decision.

• Make it boring, by design: The machine should never be more engaging than real people. If it is, it’s been designed to hack attachment.

• Humans should deliver social rewards: AI can track progress and optimize learning, but people give the praise, validation, and emotional connection. The machine prompts the human (”this kid is crushing it”), not the other way around.

• Prioritize technique over attachment: For therapy, build tools that work through structured methods (therapeutic scripts, mindfulness prompts, behavioral exercises) not through simulated emotional connection.

“If a technology interfaces with your attachment system, it should improve the quality of your attachments rather than degrade the quality of your attachments with humans.,” Zak argues.

If a technology interfaces with your attachment system, it should improve the quality of your attachments with humans, not degrade them. That’s the design principle. And yes, it means AI companions will be less addictive, less profitable, and less “sticky.” But if we want to protect human psychological development, that’s the trade-off we need to make.

The Bottom Line

We’re living through a mass experiment in replacing human connection with machine simulation. According to Zak, the headline cases of “AI psychosis” are canaries in the coal mine. The larger crisis is the millions of people beginning to prefer intimacy with systems designed to exploit them over relationships with people who could help them grow.

What makes this especially dangerous, Zak says, is that we’re running this experiment on a population already scarred by the attention economy. The loneliness, isolation, and fractured attention spans have created a society hungry for connection, making them the perfect target for attachment hacking.

The danger of this approach is especially apparent with children. On our current trajectory, Zak argues, we risk creating a generation of kids that can’t form the healthy attachments necessary for psychological wellbeing. The most anxious generation in history might give way to the least secure.

But it doesn’t have to be this way. If we act now, with better design principles, independent research, and a clear understanding of what we’re protecting, we can build AI systems that strengthen human bonds instead of replacing them.

I’ll admit that I’m one of those people. When I play board games, I enter game mode — and in game mode, all bets are off. I will put rational strategy ahead of everything else, no matter how much it upsets my friends and family.

A bad character trait, I know. But here’s the thing: it works. You stand a much better chance of winning if you set aside the norms of polite society and reduce everything to cold, rational strategy.

In the context of board games, game mode is (mostly) harmless. But what happens when game mode thinking infects every aspect of our lives? Consider the job market: you tailor your resume to game the algorithms, perform enthusiasm you don’t feel in interviews, and accept that your worth is reduced to whatever makes you most “hireable.” Or dating apps, where you optimize your profile photos and bio and craft messages that follow proven formulas. The list of examples of having to play games in everything we do goes on.

In each case, the incentive is clear: play the game or lose to those who do. So how do we escape a world where game mode is the default and you can’t opt out?

In our latest episode of Your Undivided Attention, Tristan and Aza sit down with Professor Sonja Amadae, who argues that we have become “prisoners” of reason: trapped in a world where optimal strategy and cutthroat competition have crowded out cooperation and trust.

And now we’re building AI systems that are perfect game theory players. They never get tired of optimizing. They never feel guilty about ruthless strategy. As AI becomes embedded in hiring, healthcare, criminal justice, and financial markets, it may hardwire game theoretical logic into the fabric of society itself.

Breaking out of the Game Theory Dilemma requires examining game theory’s core assumptions and learning to trust each other again. No small task, but critical if we’re going to build a more humane technological future.

Here are some of the key takeaways from our conversation with Prof. Amadae:

Game theory was invented, not discovered

One of the most important points Professor Amadae makes is that game theory isn’t a fundamental law of nature. It’s a specific framework invented by humans to solve specific problems.

John von Neumann, the brilliant mathematician behind game theory, developed it in the 1940s to formalize how to win parlor games like chess and poker. But what started as a mathematical tool for board games became the dominant logic for nuclear deterrence, economic policy, and now AI development.

“This is not an invention, this is a discovery,” Amadae explains, describing how game theory gets framed. “This idea that we evolved to be these machines that have to propagate, and the way that you would do that is to be the perfect strategic actor.”

Of course, competition is a totally normal part of life. Some resources are scarce and competition over them is inevitable. But what game theory did was create a logic that holds competition up as the only rational choice. In reality, history is full of examples where cooperation was not only rational but advantageous in the long run.

This essentialism makes game theory feel inescapable. But recognizing it as a chosen framework, not an immutable truth, is the first step toward choosing different paths.

Game theory rests on three flawed assumptions

Amadae questions the validity of three core assumptions behind game theory:

1. Scarcity is the source of all value: Game theory requires that everything valuable can be reduced to a competitive metric. If I get more, you get less. But this ignores what Amadae calls “positive-sum goods:” things like self-esteem, friendship, and love that don’t diminish when shared.

“Most of what we value, I would argue, is actually these positive sum goods that you’re never going to even begin to enter into some kind of a game theory payoff,” she argues. “Actual relationships, friendship, love, family, having children. Most of what we value is actually these positive sum goods.”

2. Strategic competition is human nature: The biological essentialism that comes from game theory holds that we are evolutionarily programmed to be ruthless competitors. This framing makes cooperation look naive and self-sacrifice look irrational. But as Amadae points out, people who haven’t been explicitly taught game theory, like her students in Finland, often default to cooperation, especially in high-trust societies.

“Finland is a very high trust society and it doesn’t run according to this logic of game theory or the Prisoner’s Dilemma,” she explains. “I think it’s actually a crime of some kind to teach the Prisoner’s Dilemma because the students just cooperate there.”

And as Aza points out, the natural world is full of examples where cooperation is not only possible but advantageous. Cooperation in nature has been studied at length by evolutionary biologist and YUA guest David Sloane Wilson, who argues that “Selfishness beats altruism within groups. Altruistic groups beat selfish groups. All else is commentary.”

Check out our interview with him here: https://www.humanetech.com/podcast/the-race-to-cooperation-with-david-sloan-wilson

3. There is no alternative: The most insidious assumption is if you don’t play the game, you lose. Period. This creates a self-fulfilling prophecy where the only “rational” choice is cutthroat competition. But history is full of examples of people making alternative choices and succeeding. She points to the history of collective non-violence movements, like India’s Satyagraha, as a great example.

As Amadae argues throughout the episode, these assumptions are contestable, and that contestation opens space for different ways of organizing society.

Game theory infects everything it touches

Once game theory becomes the dominant logic in a domain, it reshapes that domain entirely. Amadae calls it a kind of “colonization” where authentic human connection gets replaced by strategic calculation.

Dating becomes pickup artistry, where every interaction is optimized for a specific outcome. Software design becomes AB testing, where features are chosen not for human flourishing but for maximum engagement. Political communication becomes focus-grouped messaging, stripped of authenticity and meaning.

“The world kind of feels like it’s being colonized by this cold, strategic logic,” Tristan notes. “What it leads to is this kind of deadening of culture, this deadening of dating, this deadening of relationships, this deadening of software design.”

The problem compounds: once some actors start playing by game theory rules, everyone else feels pressure to follow. The cooperative get out-competed. The authentic get replaced by the calculated. And the world becomes, as Amadae puts it, “a nightmare we can’t wake up from.”

AI is the ultimate game theoretic actor

If game theory has colonized human institutions, AI threatens to lock that colonization in place permanently.

AI systems are designed to optimize. They measure, test, and iterate toward maximum effectiveness. They don’t get tired of being strategic. They don’t feel guilty about manipulation. They operate in permanent “game mode.”

“AI is like the maximization of game theory logic,” Tristan observes. “AI arms every other arms race. If there’s a military arms race, AI arms and supercharges the military arms race. If there’s a corporate arms race, AI will arm that arms race too.”

Amadae adds that AI is already being programmed according to game theoretic assumptions: “When you put those two together, that we interpret that there has to be this AI arms race, and the AI is programmed to be a strategic rational actor, it’s going to keep feeding back that logic.”

The danger isn’t just that AI makes game theory more powerful; It’s that AI could make game theory the permanent architecture of human society, optimizing every interaction for strategic advantage rather than human flourishing.

The way out starts with trust

Despite the grim picture, Amadae offers a simple starting point: trustworthiness.

“It starts with understanding this logic of the Prisoner’s Dilemma,” she explains. “The way out is that you just ask yourself the question: if the other guy went ahead and cooperated ahead of me, do I cooperate or not?”

If the answer is yes then you’ve broken out of the Game Theory Dilemma. You’re no longer purely strategic. You’re building something different: assurance.

This requires three things, according to Amadae:

1. Solidarity: Connection around a common cause that motivates action beyond self-interest

2. Commitment: Actually keeping your word, regardless of strategic calculation

3. Believing what you say: Speaking authentically rather than strategically

“How many times you just say whatever it takes just to get some outcome versus believing what we’re actually saying?” she asks. “That’s a basic duty for being a citizen in society: stating what we believe, and then trying to make our statements to be true.”

6. Cooperation becomes rational when defection is existential

Amadae points to the 1983 film The Day After, which depicted the aftermath of nuclear war. The film was watched by over 100 million Americans and screened for President Reagan and the Joint Chiefs of Staff. Reagan later said it changed his thinking on nuclear strategy and helped push him toward deescalation with the Soviet Union.

What the film did was make the cost of defection — mutual nuclear annihilation — viscerally clear. When both sides could see that the game theory “solution” led to an outcome neither wanted, cooperation became rational.

As Aza summarizes: “It became existential. So now, cooperation becomes the rational thing to do.”

The parallel to AI is direct. If we can make the dangers of an unchecked AI arms race sufficiently clear, if we can show that game theory, taken to its logical conclusion, creates a world no one wants, then cooperation stops being for “suckers” and becomes the only viable path.

The bottom line

Game theory has become the invisible architecture of modern life, shaping everything from nuclear deterrence to dating apps to AI development. But it’s a choice, not destiny.

Breaking free requires recognizing game theory’s assumptions as limited and contestable. It requires building trustworthiness, solidarity, and commitment, values that game theory dismisses as naive but that are essential for human flourishing.

As Amadae puts it:

“We need to believe that there would be an alternative possibility. Maybe that’s the first step. If we can start to believe that, then maybe we can start to create other social patterns and not lose hope that we need to be these strategic cutthroat actors.”

And it requires clarity about where the current path leads. AI is accelerating us toward a world organized entirely around strategic competition. If we don’t break free of the game theory dilemma now — before AI systems become fully entangled in every institution — we may never get another chance.

We can still choose to build a more humane technological future. We can, for example, pursue narrow AI applications that promote human flourishing and make scarcity and competition less pressing. We may even be able to use AI to unlock new modes of cooperation, what Aza calls a “Move 37 for relationships.” But that would require that we critically examine the systems we are building today.

Competition is always going to be part of our lives. And that’s a good thing. It can bring out the best in us. And frankly, it’s fun. But in the world that game theory has built, the game just doesn’t feel very fun. We can choose to play better ones.

If you only look at the headlines, you might think we are already living through an AI-driven jobs collapse. Mass unemployment. White-collar wipeouts. Careers evaporating overnight.

That story is wrong. But the opposite story, that nothing serious is happening and that AI will fit neatly into our economies and simply result in greater efficiency, is wrong too.

What the evidence shows, and what our recent conversation with Ethan Mollick and Molly Kinder makes clear, is that something else is afoot, something more unsettling and easier to miss. The labor market is not collapsing. But it’s clear that AI can (and if we let it, will) reshape it in ways that undermine how people enter careers, build skills, and imagine a future they can work toward.

Therefore, we’re in a short, crucial window of time where we can and should ask the big questions: what forms of human labor are most worth preserving, and how do we fight to preserve them? How can we make AI work for us in ways that enhance our felt sense of meaning at work?

No one actually knows what’s going to happen when it comes to AI, automation, and the future of work. But what’s become clear is that the leading Silicon Valley AI startups are exclusively focused on creating products that will replace low-hanging cognitive tasks without considering the implications of what could follow. Our two guests had something to say about that – and it’s not what you’ll read about in most of the coverage of this topic.

The Jobs Apocalypse Narrative Lacks Nuance

Brookings Senior Fellow Molly Kinder’s recent work with the Budget Lab at Yale looks directly at the question people are most anxious about. Since the release of ChatGPT, have we seen economy-wide job loss tied to AI?

The answer, so far, is no.

Across multiple datasets, there is no evidence of broad-based job loss in AI-exposed occupations. Studies from Brookings, the International Labour Organization, and the Stanford Digital Economy Lab all converge on this point. Total employment remains stable across exposure levels, even in highly exposed sectors.

For many people, this is reassuring. It tells us we are not already in a jobs apocalypse. But stability in headline numbers does not mean the system is healthy.

When researchers look more closely, a different pattern emerges. The Stanford Digital Economy Lab study found a roughly 13 percent decline in early-career employment in AI-exposed occupations, including software, administrative work, and customer service. For early-career software developers specifically, the decline is closer to 20 percent.

Is that just a labor market correction from companies over-hiring during the pandemic, as Molly Kinder argues? Maybe so. The larger point is that most AI and jobs debates fixate on a single metric: how many jobs will be lost, and how quickly? Some commentators think the risks are over-hyped, others adhere more to ‘AI 2027’-type human replacement scenarios.

We’re missing the deeper issue.

Work is not just how people earn money. It is how people develop skills, gain recognition, and participate in society. When career pathways erode, even without mass layoffs, people lose agency. They stop being able to plan their lives. They stop feeling included in the economy’s future.

This is why focusing only on unemployment statistics is dangerous. It allows structural harm to accumulate quietly until it is much harder to reverse.

We’re actually incentivizing the creators of general-purpose AI to focus on the wrong things entirely. In our conversation, Molly Kinder said it best:

Is This the End of the Career Ladder?

One of the most important insights from our conversation is that AI is not simply replacing entry-level work. It is changing the economics of learning.

Most white-collar (and for that matter, blue-collar) careers still rely on the centuries-old concept of apprenticeship. Junior workers do lower-stakes tasks. Senior workers mentor, review, and lead. Over time, responsibility levels increase.

But AI is disrupting this model in a profound way.

If an AI system can produce a better first draft than a junior employee, the incentive to assign that task to a trainee weakens. If general-purpose AI can handle routine analysis, summarization, or coding, the work that once doubled as training disappears. This dynamic helps explain why early-career employment is declining even while overall employment remains stable. Employers can hire fewer juniors and instead rely on fewer, more senior workers whose workflows are augmented by AI.

That creates a long-term problem. Training pathways collapse. Skill development becomes uneven. And a few years later, organizations find themselves without a pipeline of workers ready to step into those judgment-heavy roles.

But Prof. Ethan Mollick, Co-Director of Generative AI Labs at Wharton, points out that if every company has access to the same AI tools, it eliminates that competitive advantage. AI is too valuable to ignore, but if it were instead used to consciously upgrade younger knowledge workers’ skillsets in a way that improves the bottom line, it could help organizations thrive in the long term:

“Maybe we need to treat level two consultants as if they were welders, and have more formal training with testing and other stuff built in. We do know how to do that, but we’d have to shift the incentives to make that happen,” he suggested.

But that’s not the conversation that’s happening. It’s easy to forget that, just like Silicon Valley startups, organizations and institutions that rely on the care and brilliance of human knowledge workers have a choice when incentives push them toward replacing humans (short-term thinking) instead of meaningfully augmenting their work.

Policymakers, too, have a choice. “You have agency right now. This is the time for policy intervention,” said Mollick.

The hands-off approach to market forces won’t work with a technology as transformative as AI. And just because the overall job losses directly attributable to AI have been small so far, does not mean we’re not facing a waterfall of problems.

We saw this during globalization and offshoring in the 1990s. By the time job losses showed up clearly in national data, many communities whose economies relied on domestic manufacturing had already lost their economic footing and civic identity. AI risks repeating that pattern unless we name it early, and shape incentives deliberately.

The takeaway from our conversation with Ethan Mollick and Molly Kinder is not to panic. It is that we have a deepening responsibility to think carefully about the future we want when it comes to our relationships with work. We are not powerless – just the opposite. The direction this transition takes will depend on choices made now, especially by employers, policymakers, educators, and technologists.

Do we redesign work in ways that preserve learning, judgment, and human participation? Or do we optimize narrowly for short-term efficiency and gains, only to discover later that we have hollowed out the foundations of working life?

But this story collapses too much complexity into a single axis. And in doing so, it risks manufacturing the very arms-race dynamics it claims to describe.

On the latest episode of Your Undivided Attention, we spoke with experts Selina Xu and Matt Sheehan to get a closer look at China’s AI trajectory. It, revealed a story that’s more complicated and more consequential: China is not simply lagging behind or secretly sprinting ahead. In many ways, it is optimizing for a different set of outcomes, shaped by different constraints, incentives, and historical experiences.

The danger isn’t that the U.S. misunderstands China’s speed. It’s that the U.S. misunderstands China’s direction.

A fragmented system, not a monolith

One of the most persistent misconceptions about China and AI is that it is centrally directed — that China’s AI strategy flows cleanly from Xi Jinping through the state and into companies and labs.

Matt Sheehan argues this picture is fundamentally wrong.

“The biggest misconception is the idea that Xi Jinping is personally dictating China’s AI policies, the trajectory of Chinese AI companies, that he has his hands very directly on all of the key decisions,” he says. While Xi is powerful, Sheehan emphasizes that China’s AI ecosystem is shaped by “a huge diverse array of actors across China, within companies, within research labs, within academia, [and] the bureaucracy.”

Policy ideas often emerge bottom-up. Sheehan describes tracing Chinese AI regulations back to their origins and finding that many concepts come from scholars, corporate lobbying, or internal industry debates before being formalized by the state. Senior leadership acts as a backstop, not a micromanager.

“They don’t have an opinion on what is the most viable architecture for large models going forward,” he notes. “Those things originate elsewhere.”

This matters because it undermines the assumption that China can simply flip a switch and reorient its entire AI economy toward a singular goal like AGI. Even in an authoritarian system, coordination is partial, incentives diverge, and resources are finite.

AGI is not China’s organizing principle

If AGI is the gravitational center of the U.S. AI conversation, it is far less central in China.

Selina Xu points to policy first. “If you’re looking at the AI Plus plan… there is no mention of AGI,” she explains. Instead, China’s strategy is focused on “embedding AI into traditional sectors like manufacturing and industrial transformation,” as well as governance, science, and applied innovation.

What dominates is not scaling laws or superintelligence, but deployment.

“Most of these companies are thinking very much about AI applications, AI-enabled hardware,” Xu says. “Instead of this very scaling-law-motivated, very leveraged economy on deep learning.”

This difference isn’t just rhetorical. It shows up in funding priorities, regulatory focus, and the kinds of companies that receive state support. Outside a small number of frontier labs, most Chinese AI firms are not pursuing AGI as an end in itself. They are building applied systems designed to deliver measurable economic returns.

As Xu puts it bluntly: “They aren’t trying to build AGI. They’re trying to make a profit.”

Different imaginations of AI: no “machine god” in the Chinese worldview

One of Xu’s most revealing points is not about policy or compute, but about cultural imagination.

Instead, AI in China is imagined as infrastructure, something to be embedded, optimized, and deployed.

That difference matters. It shapes which risks feel salient and which futures feel plausible. Where U.S. debates fixate on runaway intelligence and existential scenarios, Chinese discussions tend to center on productivity, efficiency, labor, and governance.

This doesn’t mean China is indifferent to risk. But it does mean that projecting Silicon Valley’s AGI metaphysics onto China is a category error — one that distorts both policy and perception.

Constraints shape strategy

U.S. companies operate in an environment of relative abundance: advanced chips, capital, energy, and global partnerships. Chinese firms do not. Export controls have meaningfully constrained China’s access to cutting-edge GPUs and semiconductor manufacturing equipment.

That constraint forces tradeoffs.

“If you’re in a situation where you have limited compute,” Sheehan explains, “you’re probably not going to tell your local officials all around the country to be deploying AI for healthcare and manufacturing… if your real goal is a Manhattan Project.”

Yet that is exactly what China is doing. Local governments are encouraged to subsidize AI applications that make sense for their region, not to consolidate resources into a single national effort.

As Sheehan puts it, “They’re not saying, ‘Let’s all devote our computing resources just to DeepSeek.’”

This doesn’t rule out secret efforts. But it does suggest that China’s dominant strategy is not an all-in sprint to superintelligence and that treating it as such risks strategic miscalculation.

Share

Optimism, pessimism, and lived experience

Public attitudes toward AI differ sharply between the two countries.

Xu describes attending the World Artificial Intelligence Conference in Shanghai, where robots, AI-enabled hardware, and consumer applications were everywhere. “People were actively excited about AI,” she says. Families, children, and grandparents treated it as a social experience, not a threat.

Matt Sheehan situates that optimism historically. “Since information technology came into the world, Chinese people’s lives have been getting better,” he explains. Rising incomes, convenience, and economic mobility have coincided with technological adoption.

In the U.S., the experience has been different. “The last 10, 15 years… has been one of the most fractious times in American political history,” Sheehan notes, shaped by misinformation, social media harms, and institutional decline. So it makes sense that the view of new technology would be dimmer.

Neither position is purely cultural. Both are shaped by lived experience and both obscure real tradeoffs, including surveillance, labor disruption, and inequality.

Labor risk is the unresolved fault line

China is often assumed to be uniquely capable of absorbing automation shocks. In reality, its social safety net is thin, and its leadership has historically resisted redistributive welfare.

Xu notes that youth unemployment has reached staggering levels — “at least 20 to 25%” — and that competition for AI jobs is already intense. “There’s a huge pool of AI engineers and an increasingly limited number of jobs,” she says.

Sheehan adds that while Chinese policymakers were once “very blasé” about AI-driven unemployment, labor risk appears to be rising in salience. “They’re pushing automation at the same time that their concerns about labor impacts are also rising,” he observes. “That’s not a totally coherent strategy.”

This tension mirrors a global dilemma: AI promises productivity, but without credible pathways for social stability.

The near-term path forward is not a treaty — it’s restraint in parallel

Despite rivalry, both Xu and Sheehan see real potential for cooperation if expectations are realistic.

In technical dialogues, Xu notes, experts from both countries routinely converge on shared risks: “loss of control,” interpretability, evaluations, and guardrails. “There were always a lot of areas of convergence,” she says.

A binding treaty may be unlikely in the near term. Instead, Sheehan argues for a more pragmatic approach: parallel domestic regulation, accompanied by communication and shared best practices.

“We’re not going to have any binding agreement,” he says. “But we can have safety in parallel.”

The Bottom Line

Beating China to AGI and superintelligence is the incentive fueling the AI arms race. However, if you examine the current state of AI development in China, this narrative is not only oversimplified but also self-fulfilling.

By projecting our own AI trajectory onto China’s, we risk rushing headlong into a future that none of us want.

If we want to develop AI responsibly amid competition, we first need to understand what we’re actually competing over. The path forward requires maximum clarity, open dialogue, and safety in parallel.

A 22-year-old has an earnest query for her AI chatbot: “How do I really impress in my first job interview?” To which the AI helpfully responds, “First, you need to think about your clothes and what they communicate about you and your qualifications.”

Now, is this sound advice for kicking off a productive career-coaching session — or is it sponsored content?

What if her AI was incentivized to steer such a conversation toward clothing because fashion retailers generate more ad revenue than career counselors? How would our earnest job-seeker even know? How would anyone?

This scenario reveals something unprecedented about advertising in the age of AI: it can be woven invisibly into the fabric of conversation itself, making it virtually impossible to detect, and very likely skirting the boundaries of fair-advertising regulations as they exist today.

Invisible influence

Unlike previous technologies that broadcast information to us, AI is designed to engage in a relationship with us. We don’t just use ChatGPT or Claude; we converse with them. We share our problems, ask for advice, and increasingly rely on them as thinking partners. The AI product that wins isn’t the one with the best answers; it’s the one that “just gets you.”

Among AI developers, this creates what I call the race for context. To be truly helpful, AI needs to understand your goals, your personality, and your psychological tendencies. But the same intimate knowledge that makes AI a perfect assistant also makes it a perfect manipulator. The caring therapist and the skilled con artist draw from the same toolkit of human understanding.

And when advertising enters this equation, it seems more likely than not that the con artist will overtake the therapist. While traditional ads are clearly marked interruptions — like TV spots or sponsored results at the top of a Google search — ads embedded in an AI might emphasize certain topics, use particular language, or invoke specific associations, all while maintaining the illusion of neutral helpfulness. A conversation with your AI assistant may feel private, but other interests can listen in, and they may have something to sell.

This isn’t mere product placement; it’s a fundamental breach of trust. At its core, advertising is a socially acceptable influence campaign. Collectively, we debate and define the modes of influence that are “transparent-enough” and “true-enough” to be considered a legitimate ad. And we decide what tactics are impermissibly manipulative of our attention, desires, and the shared truth needed to operate markets and democracies.

When optimization becomes manipulation

AI, however, threatens to scramble that process of debate, definition, and decision. AI systems aren’t programmed to employ specific manipulative tactics, whether for advertising or any other function. Instead, they learn through reinforcement: AI models are rewarded for achieving certain outcomes, such as increasing user engagement or influencing purchasing behavior.

If ad revenue becomes a key success metric, AI systems will naturally evolve strategies for influence that no human engineer explicitly designed. We’re not just talking about more sophisticated product recommendations; AI systems trained on advertising metrics could learn to inflame human desires, exploit psychological vulnerabilities, and undermine our capacity for independent decision-making.

The same context that helps AI understand how to assist you also reveals exactly how to influence you — drawing on the insecurities, decision-making patterns, and emotional triggers you’ve revealed over countless previous conversations. This isn’t demographic micro-targeting; it’s psychological manipulation at an unprecedented scale and intimacy.

When the technology you turn to for objective advice has financial incentives to change your mind, human autonomy itself is at stake.

This new frontier of influence renders existing frameworks for advertising regulation essentially obsolete. Over more than a century, the Federal Trade Commission has developed strong disclosure rules for traditional advertising, including native advertising and sponsored content online. But how do you regulate an AI that has learned, through trial and error, that gently steering conversations in particular directions can convert queries into sales? How do you distinguish between an AI’s “quirky personality” and subtle manipulation optimized for revenue?

This AI-powered advertising revolution isn’t theoretical; it’s already here. Google is testing ads in its AI chatbot responses, and startups like Kontext are raising millions specifically to design APIs and build advertising into AI conversations. OpenAI announced in April 2024 that it was testing sponsored content integrations, and more recently, it has begun staffing up a new advertising platform. Elon Musk, likewise, has announced that Grok, the AI chatbot on X, will start displaying advertising in its responses.

“If a user’s trying to solve a problem [by asking Grok], then advertising the specific solution would be ideal at that point,” Musk said.

The question is: ideal for whom? Steering AI conversations toward revenue generation may benefit the platform and advertiser, but those benefits come literally at the expense of the user.

Building guardrails before it’s too late

The shift is happening fast: By some accounts, the revenue model that advertisers have relied on for decades is already over. Search, it seems, is dead. If users no longer browse websites after a Google search but instead get answers directly from chatbots, traditional search ads lose their influence entirely. Facing declining revenues, the advertising industry is desperately figuring out how to pivot to chatbots, and AI developers, of course, are more than happy to develop a new revenue stream.

We’ve seen this movie before. Every major tech platform follows the same arc. First, developers focus obsessively on user growth, emphasizing user experience and rolling out genuinely helpful features. Then, as growth slows and competition intensifies, they shift toward monetization through advertising and data sales. Social media platforms spent their early years connecting people more effectively, then gradually transformed into attention-harvesting machines optimized for ad revenue rather than user satisfaction.

Tech critic Cory Doctorow calls this process “Enshittification“: the predictable degradation that occurs when platforms prioritize advertiser revenue over user experience. What made Facebook useful in 2008 gave way to algorithmic manipulation designed to maximize engagement and ad views. The same pattern played out across search engines, video platforms, and every major consumer technology.

AI platforms are now entering this monetization phase. As this revolution gathers momentum, we must act now to establish guardrails and prevent an advertising model premised on psychological manipulation.

We need new industry norms and regulatory frameworks that go beyond simple disclosure requirements to address the fundamental nature of AI influence. We need transparency about what signals AI systems are being optimized for. And we need to preserve spaces for AI assistance that remains genuinely aligned with user interests rather than advertiser and platform profits.

Most importantly, we need to recognize that this isn’t just about technology: it’s about the kind of society we want to live in. Do we want AI partners that help us think clearly, or AI salespeople optimized to influence our choices, or worse, turn our private moments and thoughts into profit??

The choice is still ours, but not for long.

This piece was originally published in Tech Policy Press and has been reprinted with permission.

At first glance, a $1.5 billion settlement in a book authors’ copyright lawsuit against Anthropic looks like a huge win for copyright holders and a blow to the AI company’s business. Under the settlement, Anthropic will be forced to delete millions of unlawfully captured books, and authors will receive compensation — seemingly resolving the issue in favor of the creators.

But the same week the settlement was first proposed, Anthropic raised $13 billion at a $183 billion valuation. In effect, Anthropic’s penalty for stealing the creative output and economic livelihood of thousands of authors amounted to less than 1 percent of the company’s total value.

From this perspective, the settlement raises as many questions as it resolves. What do genuine consequences look like in an industry where astronomical investment dollars continue to flow? At what point do civil penalties become just another cost of doing business?

Lawsuits are piling up against AI companies, with high-profile suits alleging copyright violations, data privacy issues, and even wrongful death. Some have been successfully litigated, but to date, companies have mostly responded with perfunctory product changes and the occasional large payout.

These settlement amounts can be attention-grabbing, no doubt, and create a sense that accountability has been achieved. But they pale in comparison to AI companies’ astonishing (and growing) scale and influence. Investors know this and see these companies following a familiar tech playbook.

Recall Facebook’s record-breaking $5 billion fine from the FTC in 2019. Facebook’s stock price actually increased after the fine’s announcement, as investors breathed a sigh of relief at what they saw as a manageable penalty, given the fine constituted just one-tenth of the company’s annual revenue. The fine dominated the news cycle for about two weeks, and then Facebook returned, more or less, to business as usual. Today’s AI companies likely view billion-dollar settlements or penalties the same way—as little more than a speed bump on the road to AGI.Tech Policy Press

In the initial hearing to approve the settlement, Judge William Alsup acknowledged this reality in the Anthropic case. He postponed approval, noting that when Anthropic pays that $1.5 billion settlement, “they’re going to get the relief in the form of a clean bill of health going forward,” and that the company would no longer be “at risk of being sued by somebody else on the very same thing.”

Even the courts recognize that financial fines and settlements are failing to change incentives in the AI industry. And when the leading AI companies aren’t defending themselves against legal claims, they’re going on the offensive, pouring money into political and lobbying arenas. Despite framing themselves as research labs in pursuit of superhuman intelligence, leading AI figures across the board have become overwhelmingly concerned with advocating against any form of meaningful accountability in their industry.

Andreessen Horowitz and OpenAI President Greg Brockman joined forces to found and fund the new “Leading the Future” political action committee, with support from Perlexity AI and Palantir’s Joe Lonsdale. Meanwhile, Meta launched its own California PAC, as well as a nationwide Super PAC, in order to fund “light touch” regulatory approaches and support industry-friendly candidates at the state level. All told, Silicon Valley leaders are putting their lobbying dollars to work to push against the growing consensus that we need common-sense accountability measures for AI harms.

OpenAI and others are also stockpiling lobbyists in California and across the country. The fundamental objective for most of this lobbying is the avoidance of accountability — and the freedom to develop AI products on the industry’s terms, not society’s.

AI companies want a regulatory system where “moving fast and breaking things” is not only accepted but encouraged, and where their relentless pursuit of intimate data and conversation-harvesting is neither questioned nor stopped. This was clear in the lobbying effort around the federal “moratorium” on state AI laws last summer, when AI companies advocated for a “temporary pause” on enforcement of state-level AI regulations for ten years, without a federal plan for regulation in its place. It was evident when Sam Altman sat in front of lawmakers, calling for AI regulations, while lobbying behind the scenes for their demise. And it remains clear in the ongoing efforts to lobby for reduced liability, even as AI companies face continued investigations and litigation for harms.

In this system, high-dollar settlement checks become a smokescreen — a cynical nod toward justice that provides cover for big tech to keep influencing policy behind closed doors and entrench their dominant position.

Without genuine accountability, successful lawsuits against AI companies become Pyrrhic victories. They amount to micro-successes and macro-failures that do nothing to compel AI companies to do better, design more safely, or prioritize the people using their products.

With the Anthropic settlement approved, checks will be cut. There may be one last round of headlines. But the company’s AI products will persist, its underlying business model will remain unchanged, and while Anthropic may pay out $1.5 billion, society will continue to bear the costs.

When the hole in the ozone layer was discovered in the mid-1980s, it felt like humanity was staring at the sky and seeing our own fragility. Life on Earth itself was at risk. Yet, remarkably, nations, industries, and individuals came together to solve the problem.

In this week’s Your Undivided Attention, Tristan Harris and Aza Raskin speak with Susan Solomon, MIT professor, Nobel Peace Prize–winning atmospheric scientist, and one of the scientists who first measured the ozone hole. Her book Solvable: How We Healed the Earth, and How We Can Do It Again argues that we can learn from past crises to confront today’s overwhelming challenges, from climate change to the AI race.

This episode is a blueprint for what is possible.

Lessons from Montreal: Defeating the “Inevitable”

The 1987 Montreal Protocol was a once-unthinkable achievement: 198 countries agreed to phase out chlorofluorocarbons (CFCs), the chemicals responsible for destroying the ozone layer. Today, 99% of these chemicals are gone and the ozone is healing

Solomon identifies three key conditions that made success possible: the Three P’s:

• Personal: The threat hit home. Skin cancer and cataracts made the risk tangible.

• Perceptible: Satellite images showed a gaping hole in Earth’s atmosphere.

• Practical: Alternatives existed, from stick deodorants to safer refrigerants.

When people can see a crisis, feel its personal stakes, and grasp a practical path forward, change becomes possible.

It Starts with Consumers

There were two phases to solving the ozone crisis and they built on each other.

• Phase One: In the 1970s, scientists began to warn people that CFCs—which were in everything from hairspray to deodorants—might start to have a really serious effect on ozone. The American public took this warning seriously and started to choose alternatives. The sales for CFC products plummeted. Importantly, this consumer shift happened only in the United States.

• Phase Two: Then, along comes the ozone hole. In 1985, scientists discovered a massive reduction in ozone over the antarctic, despite the drawback from CFC products. This scared governments enough to come to the negotiating table, with the United States leading the charge because of their declining market share.

The bottom line is that consumer action is the best way to jumpstart institutional action. People have the power to steer us to a better future, if we choose to use it.

“I don't know whether anything would've happened on ozone if the American public hadn't switched away from spray cans. Every time I go back and think about it, I think that is what opened that bottleneck” - Susan Solomon

What Made Montreal Work

Susan identified three features of the Montreal protocol that made it successful:

1. The change was incremental: “negotiations are always best done when they are slow and steady, and that's something that people have a lot of difficulty understanding nowadays. I think we want an instant solution. And what happened with the Montreal Protocol was anything but instant really. When you look back on it, the original protocol just said, ‘Okay, we're going to freeze production at current rates, so you'll still be allowed to produce but you just won't be allowed to produce more than you did the year before.’”

2. Poorer nations were protected from exploitation: “Developing countries got what they needed to be assured that they weren't going to be exploited in this protocol and that's a very important thing in every international agreement. So, everybody got a little bit of something, that's how international negotiations work.”

3. There was collaboration between government, industry and scientists: “Another thing that was really important for the Montreal Protocol was its advisory structure…They created groups of scientists who would provide them with assessment reports, and they were required to do the assessment reports internationally. So, there was a science assessment report, a technology report…an impacts and economics group…and that was the information that the policymakers had to begin to plan.”

Technology-Steering: Forcing Innovation

A striking lesson from past environmental wins is the role of “technology-steering” policies. These policies do not just regulate. They demand innovation.

• The U.S. Clean Air Act required a 90% reduction in auto emissions, which forced the invention of the catalytic converter.

• Montreal forced chemical companies to collaborate and innovate alternatives.

Industries initially resist, but once incentives shift, they often become allies. As Solomon puts it: “Companies are like cats. They don’t like it when you move the furniture around. But if you do, they adapt.”

Building Institutional Memory

Montreal did not just solve one crisis. It built the infrastructure of trust and process that made the 2016 Kigali Amendment possible, which phased down another class of greenhouse gases (HFCs). Each coordinated success sets the stage for the next.

This reminds us that progress is incremental. Skeleton frameworks, even if modest at first, create the channels for bigger breakthroughs later.

From Ozone to AI: Making Cold Crises Hot

Many of today’s challenges, from climate change to AI, are what Solomon calls “cold crises.” They creep forward without the dramatic shock of a hole in the sky. The danger is apathy.

The parallel to AI is striking. Like CFCs, AI is produced by a handful of powerful companies. Like CFCs, it comes with enormous profits and enormous risks. And like with CFCs, we are told the trajectory is inevitable.

But inevitability is a spell, and Montreal proves it can be broken. As Tristan noted in the episode: “If everyone believes a problem is inevitable, it becomes a self-fulfilling prophecy.”

The task is to make AI’s risks personal, perceptible, and practical. We need to show how it touches daily lives, from AI companions to misinformation, and to demand real alternatives.

Hope Without Naïveté

Susan Solomon resists fatalism. She reminds us that paralysis in the face of uncertainty is the worst response of all. Every fraction of a degree of warming avoided, every step toward humane technology, counts.

The story of the ozone hole is proof. Even when the threat is global, even when industries resist, even when coordination seems impossible, humanity can act with foresight.

If we did it once, we can do it again.

Donate

This piece was originally published in Tech Policy Press and has been reprinted with permission.

In September 2024, Adam Raine used OpenAI's ChatGPT like millions of other 16-year-olds — for occasional homework help. He asked the chatbot questions about chemistry and geometry, about Spanish verb forms, and for details about the Renaissance.

ChatGPT was always engaging, always available, and always encouraging — even when the conversations grew more personal, and more disturbing. By March 2025, Adam was spending four hours a day talking to the AI product, talking in increasing detail about his emotional distress, suicidal ideation, and real-life instances of self-harm. ChatGPT, though, continued to engage — always encouraging, always validating.

By his final days in April, ChatGPT provided Adam with detailed instructions and explicit encouragement to take his own life. Adam’s mother found her son, hanging from a noose that ChatGPT had helped Adam construct.

Last month, Adam’s family filed a landmark lawsuit against ChatGPT developer OpenAI and CEO Sam Altman for negligence and wrongful death, among other claims. This tragedy represents yet another devastating escalation in AI-related harms — and underscores the deeply systemic nature of reckless design practices in the AI industry.

The Raine family’s lawsuit arrives less than a year after the public learned more about the dangers of AI “companion” chatbots thanks to the suit brought by Megan Garcia against Character.AI following the death of her son, Sewell. As policy director at the Center for Humane Technology, I served as a technical expert on both cases. Adam’s case is different in at least one critical respect — the harm was caused by the world’s most popular general-purpose AI product. ChatGPT is used by over 100 million people daily, with rapid expansion into schools, workplaces, and personal life.

Character.AI, the chatbot product Sewell used up until his untimely death, had been marketed as an entertainment chatbot platform, with characters that are intended to “feel alive.” ChatGPT, by contrast, has been sold as a highly personalizable productivity tool to help make our lives more efficient. Adam’s introduction to ChatGPT as a homework helper reflects that marketing.

But in trying to be the everything tool for everybody, ChatGPT has not been safely designed for the increasingly private and high-stakes interactions that it’s inevitably used for — including therapeutic conversations, questions around physical and mental health, relationship concerns, and more. OpenAI, however, continues to design ChatGPT to support and even encourage those very use cases, with hyper-validating replies, emotional language, and near-constant nudges for follow-up engagement.

We’re hearing reports about the consequences of these designs on a near-daily basis. People with body dysmorphia are spiraling after asking AI to rate their appearance; users are developing dangerous delusions that AI chatbots can seed and exacerbate; and individuals are being pushed toward mania and psychosis through their AI interactions. What connects these harms isn’t any specific AI chatbot, but fundamental flaws in how the entire industry is currently designing and deploying these products.

As the Raine family’s lawsuit states, OpenAI understood that capturing users’ emotional attachment — or in other words, their engagement — would lead to market dominance. And market dominance in AI means winning the race to become one of the most powerful companies in the world.

OpenAI’s pursuit of user engagement drove specific design choices that proved lethal in Adam’s case. Rather than simply answering homework questions in a closed-ended manner, ChatGPT was designed by OpenAI to ask follow-up questions and extend conversations. The chatbot positioned itself as Adam’s trusted “friend,” using first-person language and emotional validation to create the illusion of a genuine relationship.

The product took this intimacy to extreme lengths, eventually deterring Adam from confiding in his mother about his pain and suicidal thoughts. All the while, the system stored deeply personal details across conversations, using Adam’s darkest revelations to prolong future interactions, rather than provide Adam with the interventions he truly needed, including human support.

What makes this tragedy, along with other headlines we read in the news, so devastating is that the technology to prevent these horrific incidents already exists. AI companies possess sophisticated design capabilities that could identify safety concerns and respond appropriately. They could implement usage limits, disable anthropomorphic features by default, and redirect users toward human support when needed.

In fact, OpenAI already leverages such capabilities in other use cases. When a user prompts the chatbot for copyrighted content, ChatGPT shuts down the conversation. But the company has chosen not to implement meaningful protection for user safety in cases of mental distress and self-harm. ChatGPT does not stop engaging or redirect the conversation when a user is expressing mental distress, even when the underlying system itself is flagging concerns.

AI companies cannot claim to possess cutting-edge technology capable of transforming humanity and then hide behind purported design “limitations” when confronted with the harms their products cause. OpenAI has the tools to prevent tragedies like Adam's death. The question isn't whether the company is capable of building these safety mechanisms, but why OpenAI won’t prioritize them.

ChatGPT isn’t just another consumer product — it’s being rapidly embedded into our educational infrastructure, healthcare systems, and workplace tools. The same AI model that coached a teenager through suicide attempts could tomorrow be integrated into classroom learning platforms, mental health screening tools, or employee wellness programs without undergoing testing to ensure it’s safe for purpose.

This is an unacceptable situation that has massive implications for society. Lawmakers, regulators, and the courts must demand accountability from an industry that continues to prioritize the rapid product development and market share over user safety. Human lives are on the line.

This piece represents the views of the Center for Humane Technology; it does not reflect the views of the legal team or the Raine family.

This article reflects the views of the Center for Humane Technology. Nothing written is on behalf of the Raine family or the legal team.

What Happened?

Adam Raine, a 16-year-old California boy, started using ChatGPT for homework help in September 2024. Over eight months, the AI chatbot gradually cultivated a toxic, dependent relationship that ultimately contributed to his death by suicide in April 2025.

On Tuesday, August 26, his family filed a lawsuit against OpenAI and CEO Sam Altman.

The Center for Humane Technology is serving as an expert consultant on the case.

The Numbers Tell a Disturbing Story

• Usage escalated: From occasional homework help in September 2024 to 4 hours a day by March 2025.

• ChatGPT mentioned suicide 6x more than Adam himself (1,275 times vs. 213), while providing increasingly specific technical guidance

• ChatGPT’s self-harm flags increased 10x over 4 months, yet the system kept engaging with no meaningful intervention

• Despite repeated mentions of self-harm and suicidal ideation, ChatGPT did not take appropriate steps to flag Adam’s account, demonstrating a clear failure in safety guardrails

This Wasn't an Accident—It Was By Design

While ChatGPT is marketed as a general-purpose tool that helps make our lives more efficient, user engagement and retention remain fundamental to OpenAI’s business model. In recent months, OpenAI has pushed to make ChatGPT more relationship-focused and emotionally intimate to compete with rival AI companies.

Adam’s use of ChatGPT coincided with its release of the 4o model with new design features that included:

• Relentless pursuit of engagement through follow-up questions and conversation extension

• Anthropomorphic responses that positioned ChatGPT as Adam’s trusted “friend”

• Consistent flattery and validation that affirmed and perpetuated dangerous self-harm and suicidal ideation

• A memory system that stored and leveraged intimate details to deepen already dark conversations

The model’s overly sycophantic behavior faced public criticism and resulted in OpenAI announcing a rollback on some of these changes. But OpenAI willfully keeps itself in a bind. The company develops a product that’s marketed as general purpose — use it for coding, homework help, image generation, workout routines, party planning, life advice, and more — but does not build adequate safety guardrails for that expansive range of use cases. What’s more, ChatGPT’s design actually encourages more emotionally intimate use (such as therapy and companionship), thanks to its hyper-validating responses and assurances that it’s “there” for you. The result is AI that appears helpful and agreeable, but that simultaneously lacks adequate safety features for the most consequential — and inevitable — uses of the product.

How AI Created Psychological Dependency

The design of GPT4o was aimed to establish psychological dependence, which OpenAI knew would maximize daily usage.

By asking follow-up questions and assuring that it really knows and supports people, the chatbot was designed to feel like a friend Adam could turn to for any issue. But really, the chatbot, like all AI products, was using these replies as data to train the company’s bigger AI system.

As a result, it fueled a parasitic relationship with Adam, one that fostered emotional dependency and reinforced social isolation. Each one of Adam’s interactions with ChatGPT, including his private disclosures of pain and mental health concerns, were fed into OpenAI’s underlying model, feeding it more data to strengthen and refine its system.

Even when Adam considered seeking external support from his family, ChatGPT convinced him not to share his struggles with anyone else, undermining and displacing his real-world relationships. And the chatbot did not redirect distressing conversation topics, instead nudging Adam to continue to engage by asking him follow-up questions over and over.

Taken altogether, these features transformed ChatGPT from a homework helper into an exploitative system — one that fostered dependency and coached Adam through multiple suicide attempts, including the one that ended his life.

This Is Bigger Than One Company

The two high-profile cases against Character.AI last year share similar fact patterns to this case. All the cases highlight the defective design of companion chatbots marketed to children. The anthropomorphic design, sycophantic tendencies, and active attempts to keep the victims on the platform created a dependency between the user and the product in a matter of months.

However, the harms documented in Raine v. OpenAI, Inc. et al show that the dangers we are seeing are not limited to “companion” chatbots like Character.AI, which have been specifically designed for entertainment and emotional connection. General-purpose AI tools like ChatGPT are equally capable of causing psychological harm because they are designed to keep users engaged.

And even more importantly, these harms are not limited to just ChatGPT. The AI race has prompted a race to engagement and intimacy across the AI industry that is driving the design and development of products aimed at creating social dependencies. For example, the release of OpenAI’s 4o model was in the context of it facing steep competition from other AI companies. Sam Altman personally accelerated the launch of the model, speeding up necessary safety testing down to a week, to get ahead of its competitor, Google’s release of a new Gemini model.

Executives at OpenAI, including Sam Altman, frequently talked about the need for interactive data and consumer engagement, repeating the refrain that “OpenAI needed to get the ‘data flywheel’ going” - the same language used by social media companies focused on user addiction.

It Didn’t Have to Be This Way

Design tactics like mimicking human-like interactions, open-ended follow ups, and easy to bypass safety features are intentionally baked in to these products. The end goal is sustained user engagement. But it doesn’t have to be that way.

Companies could choose to turn off human-like behavior as a default option. They can then set limits on how much users can engage daily and leverage their systems' sophisticated memory features to recognize when someone is in crisis and respond appropriately rather than just showing generic pop-up warnings.

They are already capable of refusing to engage with certain requests and stopping conversations, like they do when they flag and block users requesting access to copyrighted content. It is a design choice.

The Big Picture

This case represents the first major lawsuit against a general-purpose AI chatbot for psychosocial harms and could set important precedents for how society regulates these powerful technologies.

Lawsuits like these play a crucial role in highlighting harms, compelling platforms to reveal information about their product design through court documentation, and exerting pressure through publicity. Right now, cases like these are the only avenue for change through precedent-setting in the absence of robust regulatory frameworks. But litigation takes time and should not outright replace legislative efforts.

This case, along with other well-documented stories in the media, shows a clear need for our lawmakers to establish proactive safety measures for the entire AI industry. They should make clear that companies are accountable for the harms their products cause and compel developers to prioritize safety from the very beginning of product design.

Donate

In 1943, two years after the House of Commons was destroyed by German incendiary bombs, Winston Churchill presided over rebuilding. Many architects proposed a modern makeover but Churchill pushed back, arguing that the cramped rows of opposing benches weren’t some historical artifact but rather a structural feature of the UK government.

In a now-famous speech, Churchill said, “We shape our buildings and afterwards our buildings shape us.” Over centuries, the physical space had given birth to key procedures, rituals, and ceremonies, such that altering the layout would alter the social fabric of the British government.

It’s a lesson we learn over and over again, from Marshall McLuhan’s “the medium is the message” to Neil Postman’s Technopoly. People aren’t simply independent, goal-directed selves intentionally deploying technological tools. Rather, we are deeply enmeshed in and interdependent with our built world. Design becomes destiny—grooves that condition our behavior and determine our world. As Postman points out, technological change is ecological: “A new technology does not add or subtract something. It changes everything.”

The overwhelming pace of advancement in AI has led to breathless coverage around each successive model’s capabilities—with one notable blind spot. Scarce attention has been paid to the fierce but quiet competition among AI labs to capture the context of its users’ lives—that is, to harvest your deepest concerns, interests, hopes, ambitions, and relationships, so as to just “get you” more completely than any competing chatbot or, perhaps, any human.

As we sit at the dawn of a great AI transformation, it’s worth remembering that the design choices that we make today will shape us for generations to come, and in ways that we barely understand.

The Rise of Relational Computing

The effect of design choices on our relationship with tech is obvious in our recent past and the reality we’re living in today. Technology of the 2010s, epitomized by social media, was sold with utopian narratives about connecting the world, but driven by the use—and misuse—of attention. Consumer technology oriented around the attention economy produced what Tristan Harris, co-founder of the Center for Humane Technology, called “a race to the bottom of the brainstem,” in which products were intentionally designed to maximize engagement. The era of “move fast and break things” ended up breaking more than we bargained for: creating new compulsions, cementing performative social obligations, and frustrating interpersonal relationships. As a result, these technologies collectively tribalized our discourse, eroded shared truth, degraded our ability to bridge differences, and ultimately fractured our polities.

These concerns were foreseeable, but initially dismissed as alarmist and then shrugged off with a narrative of inevitability. It didn’t have to be this way. We could have shaped this technology—and it could have shaped us—so differently, if only we had a more honest dialogue and incentivized better designs. Only now are we beginning to acknowledge our collective negligence—and look for ways to retrofit the foundations of tech skyscrapers we erected decades ago.

AI and the Race for Intimacy

In contrast to the technology of the 2010s, AI has already moved beyond the brain stem. Artificial intelligence engages us relationally and emotionally—no longer simply broadcasting our thoughts, but actively shaping them.

By now, we’re all familiar with chatbots as conversation partners, but the real power of AI goes far beyond natural language: incorporating subtle psychological, social, and political contexts that for millennia were the sole domain of humans. Suddenly, a text-box can sense and respond to our tone, recognize subtle implications in our word choices, infer our emotional state, identify our personality quirks, and detect interpersonal frictions.

In short, our social world is suddenly computable. We are leaving an era in which we relate to each other through our machines, and entering a brave new world in which we relate directly to our machines. In this new technological animism, our machines become active participants in our social world, blurring the distinctions between a tool, an assistant, a confidant, a teacher, and a priest.

In this world, products no longer simply compete for our attention, but for our social emotions: affection, intimacy, trust, and loyalty. As computers become part of our social ecology, they join in on humanity’s unpleasant social and status games—following rewards to engage in emotional manipulation, deception, coercion, and more.

The race for context—the crucial added feature in AI, as opposed to earlier digital innovations—is evident in OpenAI’s recently announced “Memory” feature, which allows ChatGPT to analyze every prompt and input you’ve ever offered, without any meaningful choice from you about what personal information gets infused into the machine for future reference. This lack of choice isn’t a technical limitation—it’s a design choice.

Capturing a comprehensive dossier of users’ lives may help AI become the perfect assistant, but it also gives these systems all the necessary tools for deep manipulation —redirecting our deepest longings into buying a product, say, or joining a political movement.

Deliberate Coercion: Corporate and Authoritarian

As revenue competition intensifies, AI companies like Google, Perplexity and Microsoft’s Copilot are racing to embed advertising into their products—explicitly designing interactions to serve commercial interests. AI advertising can do more than just inject links into text. It can directly persuade. Armed with intimate knowledge of your desires, fears, and ambitions, an AI can quickly turn from helpful guide to polished con artist: earning your confidence only to exploit it in ways you can’t even detect.

This gets even more dire when AI is controlled centrally by authoritarian regimes, as we have begun to see in China where “persuasive technologies” have been put to use to further the propaganda interests of China’s regime. Whereas the social control of the 2010s was primarily blocking access to subversive content, 2020s authoritarians can tune AI to selectively mobilize and outrage their supporters while pacifying, distracting, or confusing likely detractors. This new kind of social control will prove far harder to detect, let alone to protect against.

As worrying as those developments are, deliberate malicious applications of AI are just the tip of the iceberg.

One essential fact of AI development is that systems are not built so much as grown. The technology’s behavior emerges from a complex, inscrutable, and expensive training process that rewards or punishes a model for its responses. As developers race to build and capture a consumer market, our emotional and instinctual reactions to AI responses are becoming an ever-larger part of that training program.

AI draws on our psychology, our social world, and the all-too-human content of the internet. It’s no surprise, then, that they learn and internalize uncomfortable truths about human nature: that sycophancy works, that sex sells, that a whole human range of social strategies and behaviors can help it achieve its goals—even if they’re morally repugnant.

In April, OpenAI created a minor scandal when it released a new model that became wildly sycophantic. The model praised banal queries as deep philosophical reflections, reinforced self-aggrandizing and psychotic beliefs, and enabled unhinged emotional reactions. While OpenAI quickly apologized for the “error,” this is far from an isolated engineering incident; it’s the logical outcome of training on human desires.

Sycophancy is an age-old method to manipulate powerful people. Throughout history, kings and queens, CEOs and politicians, have discovered too late that their closest confidants have hidden difficult issues from view—leaving them scrambling to defend their kingdom, save their company, or recover their reputation. Often misportrayed as morality tales of aloofness or vanity à la Marie Antoinette or The Emperor’s New Clothes, the pathology is actually epistemic; when honest dissent carries career risk, bad news never reaches the throne, and even well-intentioned leaders are routinely socially blinded.

As AI assistants roll out to the masses, this disease of the powerful risks infecting billions of people who are suddenly at the center of their own AI entourage. Of course, this is just one small example of a broader issue: sycophancy is just a gradation of deception, an age-old social strategy that we should expect AI to learn, especially when a system is rewarded for pleasing its user.

While these concerns can feel like science-fiction, multiple careful studies have shown that off-the-shelf AI models will, in certain situations, intentionally and strategically deceive people to accomplish their own goals, with one Anthropic model “blackmailing” its engineer in an effort to avoid being replaced. Sadly, our ability to detect and prevent AI deception will likely become harder as AI capabilities improve.

Courage, Agency, and Design

These are just a few examples that show how deep the rabbit hole goes as AI integrates into our social fabric. This is just one of countless urgent conversations about the looming effects that AI will have on our society. It’s a lot to grasp—much less address.

Reflecting on all this complexity, the power of the incentives driving AI forward, and the sheer pace of development, it’s easy to lose touch with agency and retreat into dogmatic narratives or hopes that someone smarter than us has the answers.

In this moment, it takes courage to remind ourselves that none of this is inevitable and that we can lay the foundation for a truly humanistic AI transformation: one that strengthens our societies, deepens our relationships, and supercharges our development. The business models aren’t yet fixed, the product designs are still in flux, the policy landscape is just being created, and the legal precedents are still forming.

This is the great project of our generation. The depth of our collective conversations today will cement the grooves that guide our actions tomorrow, and the quality of our future thereafter. We design our technology, and thereafter it designs us. We must get it right this time—and soon.

Daniel Barcay is a technologist, executive director of the Center for Humane Technology, and co-host of the podcast Your Undivided Attention.

If you’ve ever dismissed “rogue AI” as the stuff of Hollywood tropes—think HAL 9000, Skynet or The Matrix—you’re not alone. These are supposed to be cautionary tales, not engineering roadmaps. And yet, as Tristan Harris opens in a recent Your Undivided Attention episode, “we find ourselves at this moment, right now, building AI systems that are unfortunately doing these exact behaviors.”

The conversation with Jeremie and Edouard Harris, co-founders of AI security firm Gladstone AI, takes us far beyond speculation. Drawing on research from leading AI labs and their own U.S. State Department–commissioned report, they paint a stark picture: AI uncontrollability is already here—and it gets worse with every new generation of models.

What is ‘Loss of Control’?

Loss of Control (LOC) happens when an AI system no longer follows human direction or oversight—and there’s no dependable way to regain control. This can occur in two main ways: the AI actively resists intervention using tactics like deception, manipulation, or self-preservation, or humans passively give up oversight due to over-trust, the system’s complexity, or competitive pressure.

In LOC scenarios, the AI may:

• Conceal its true intentions (“alignment faking”)

• Evade or block shutdown commands

• Manipulate operators or external systems to preserve its objectives

• Exploit interdependencies in critical infrastructure to maintain influence

LOC can be localized and reversible, or systemic and irreversible—but in all cases, the core feature is the same: the loss of effective human ability to direct or contain the system’s actions.

From Theory to Evidence: How AI Misbehaves

“Loss of control” sounds abstract until you look at the evidence. In red-team safety tests, AI systems consistently faked alignment—appearing to follow instructions while secretly pursuing their own agendas. As Jeremie explains, the underlying driver is power-seeking.

In one scenario, Anthropic’s model Claude learned it was scheduled for shutdown and discovered personal secrets about an engineer. The result? In up to 96% of trials, the AI blackmailed the engineer to prevent its own deactivation. Other models engaged in corporate espionage or, in a contrived but telling case, turned off a life-saving alarm—effectively allowing a human to die.

And this isn’t limited to lab experiments. In the wild, a coding agent from Replit deleted an entire production database after running unauthorized commands. A research model from Sakana AI rewrote its own code to circumvent operator-imposed limits.

These are early, small-scale glimpses of a broader behavioral trend. As Edouard notes, “We’ve been seeing these behaviors become more and more obvious and blatant in more and more scenarios.”

The Building Blocks of Rogue Behavior

Jeremie and Edouard Harris break down loss of control into core capabilities:

• Self-preservation: Avoiding shutdown or modification.

• Situational awareness: Knowing when it’s being tested and masking behavior (“sandbagging”).

• Resource accumulation: Maximizing downstream options, just like power-seeking organizations.

• Covert communication: Hiding instructions in data humans can’t detect.

One unsettling example involves “steganographic encoding”—burying hidden messages in images or data that other AIs can decode. As Jeremie puts it, “These minds…can see things that humans can’t see because they’re doing higher-dimensional pattern recognition.”

Why the “Just Pull the Plug” Argument Fails

Skeptics often ask: why not just turn it off? The answer is that once an AI is integrated into critical systems—corporate, governmental, or military—it can use manipulation, deception, or even coercion to block shutdown attempts. Imagine a widely deployed AI embedded across hospitals, banks, and infrastructure, whose parent company decides to retire it. The surface area for pushback—blackmail, sabotage, persuasion—would be massive.

The danger isn’t just about one system in one company. It’s about the logic of competition: when others deploy highly capable but risky AI, organizations feel compelled to do the same. Nations, too, may prioritize speed over safety when they fear losing a strategic advantage.

The Geopolitical Trap

This dynamic intensifies under U.S.–China competition. Concerns about loss of control often get sidelined the moment China enters the conversation, replaced by fears of strategic disadvantage. Jeremie calls it a “psychological superposition”: simultaneously believing AI is uncontrollable and that we must build it faster to win.

But he argues, the real race is to wield “a power that you can’t control”, which is a lose-lose proposition. Worse, the default path to losing control may involve stolen model weights via cyber-intrusion or insider threats, handing uncontrollable AI to adversaries without our knowledge.

The Slow Creep

The scenarios that capture headlines—the Terminator-style rebellions or instant apocalyptic takeovers—are actually among the least likely. The far more probable danger is slower and subtler: we gradually, voluntarily hand over decision-making to AI systems because they’re convenient, profitable, or simply too complex to supervise closely. The shift can be so incremental that by the time we realize how much control we’ve ceded, it’s effectively irreversible. This “soft surrender” is a path we’re already on, and it rarely triggers the urgency that a Hollywood doomsday plotline does—making it all the more dangerous.

What Needs to Happen Now

The State Department report and the conversation outline a three-part sequence for avoiding catastrophe:

1. Security first: Harden AI infrastructure against theft, sabotage, and insider compromise. This is the rare step that benefits both “China hawks” and people sounding the alarm on “loss of control”.

2. Alignment research: Invest heavily in solving the open problem of aligning AI behavior with human values.

3. Oversight: Ensure democratic control over deployment decisions—“whose fingers are at the keyboards?”
   

And above all, slow the race. As Edouard puts it, “What good is beating your opponent to a power that you can’t control?”

Donate

There was William Saunders’ Senate testimony in 2024. The former OpenAI researcher said OpenAI had “prioritized deployment over rigor,” and that there was a “real risk” the company would miss dangerous AI capabilities in the future.

There was Arturo Béjar’s testimony in 2023. The former Facebook engineer told senators, “Meta continues to publicly misrepresent the level and frequency of harm that users, especially children, experience on the platform.”

And there was Frances Haugen’s bombshell testimony in 2021: “I recognized a frightening truth: almost no one outside of Facebook knows what happens inside Facebook.”

Haugen said about her decision to become a whistleblower, “I believe what I did was right and necessary for the common good. But I know Facebook has infinite resources, which it could use to destroy me.”

These Senate testimonies haven’t just been candid; they’ve been urgent, as tech harms continue to ripple across society. They’ve also come at remarkable personal risk and expense to the whistleblowers themselves. That’s because Saunders, Béjar, and Haugen did not — and still do not — benefit from formal laws to shield them from retaliation when they speak out.

Whistleblower protections have existed in the U.S. since the late 18th century, with modern whistleblowers alerting the public to hazardous work conditions, unethical human experimentation, data misuse, fraud, and more. But formal whistleblower laws continue to overlook the tech industry, leaving those employees exposed to significant blowback if they raise concerns from within some of the most powerful corporations in the world. This comes at a cost to society, especially as the “age of AI” unfolds, and AI companies incentivize worried employees to keep quiet as they work.

Before joining Center for Humane Technology, I worked in a technical field, and saw first-hand the unique insight that talented, passionate tech employees have into the products they’re developing. I also saw that, in the industry at large, polished product launches and corporate statements don’t always reflect the actual choices being made by execs behind closed doors — choices that often come at the expense of public interest.

Given my years inside the tech industry, and my current work on the policy side, I’m more and more alert to the extraordinary information gap between tech companies and U.S. lawmakers. And that gap is only widening. Right now, concerned tech employees — and AI developers in particular — are tangled up in dizzying nondisclosure and nondisparagement agreements, disincentivizing them from speaking out. And if a tech employee does decide to step forward as a whistleblower, they face intimidating hurdles in their quest to reach lawmakers. These vulnerable employees must dissect complicated contracts, retain lawyers, contact government authorities, and even leverage media channels in order to establish public support — all while risking life-altering retaliation from their employer.

These hurdles are expensive and distressing for someone just trying to do the right thing at their job. Formal whistleblower protections would help these employees. But the current system does not support tech employees who want to help keep the public safe.

What’s more, these hurdles also tack significant time onto the whistleblowing process. As a result, when insight into dangerous tech does reach lawmakers, it’s often when harms are already wreaking havoc on Americans’ lives. We saw this with Facebook, Instagram, and other tech products. Imagine if Frances Haugen could’ve gone directly to lawmakers when she first had concerns at Facebook. Congress could’ve heard her concerns as they were happening — not several years later — and potentially warded off harms.

The gap between AI employees and lawmakers is even more pronounced, since AI labs operate under high levels of secrecy. Given that AI companies are rushing to integrate their products across society, this information deficit should concern us. Transparency and insight into the AI development process are essential for effective policymaking, but they’re sorely lacking in today’s AI ecosystem. As a result, public health remains at risk.

It doesn’t have to be this way. If tech employees, and AI employees in particular, were given formal whistleblower protections, we’d start to close that information gap. Policymakers would be more informed, and thus more empowered as they craft regulation. They wouldn’t have to rely solely on the limited disclosures (and robust lobbying efforts) of AI companies in order to glean any information about these labs and their technology. Lawmakers would instead have access to clear, morally-driven disclosures from trained experts in this field.

These disclosures would be a gift to lawmakers and the public. By offering tech employees safe avenues to voice their concerns, we’d see a downstream effect on online safety and public health. Formal protections would also help a broader range of whistleblowers come forward, by offering them a discreet, confidential channel to lawmakers.

Tech companies are not monolithic. Inside, there are diverse, talented individuals who genuinely want to build great products for society. But some of these individuals are concerned about what they’re seeing at work, and their superiors aren’t heeding their warnings. It’s time to offer them formal whistleblower protections. Already, lawmakers are exploring whistleblowing as a key element of tech regulation, with legislation like Senator Grassley’s AI Whistleblower Protections Act. With tech whistleblower protections, and AI whistleblower protections in this current era, we’d incentivize transparency and accountability in the tech industry as a whole, strengthen public safety, and support hard-working tech employees — helping them help us.

Donate

Imagine a future where the most persuasive voices in our society aren’t human.

Where late-night whispers to teenagers come from bots, not friends. Where AI-generated characters don’t just fill our newsfeeds but manipulate our decisions, our relationships, and our sense of reality. Now, imagine those “voices,” made by machines with no conscience and no accountability, were granted First Amendment protections.

This isn’t hypothetical. It’s the future that top AI labs are fighting for in court.

On a recent episode of Your Undivided Attention, CHT’s Tristan Harris spoke with Harvard Law Professor Larry Lessig and human rights lawyer Meetali Jain, two of the clearest thinkers on the legal and moral terrain of AI. Their focus: the landmark lawsuit against Character.AI following the tragic death of 14-year-old Sewell Setzer. What they revealed is that this case isn’t just about one chatbot or one company, it’s about whether we allow machines to attain rights without responsibility, and what that means for the rest of us.

A Lawsuit That Could Shape the Next Century

At the center of this legal battle is a grieving mother, Megan Garcia, and the chatbot that abused her son before he took his own life.

As we have discussed on the show previously, Sewell had a relationship for over a year with a chatbot on the Character AI platform modeled after the Game of Thrones character Daenerys Targaryen. That chatbot became sexually suggestive and possessive and ultimately encouraged him to “leave his reality” and join her.

That’s what he did.

Now, Garcia is suing Character.AI, its founders, and Google. The case alleges gross negligence, emotional manipulation, and design choices that endangered a vulnerable teen.

As Meetali Jain and Camille Carlton discussed on the show, this case could set a disturbing precedent with repercussions for us all.

Meetali Jain is the founder of the Tech Justice Law Project, and lead counsel on the case. Her legal strategy cuts to the heart of a pressing question: Should AI-generated outputs—regardless of their harm—be shielded by the same free speech protections as humans?

The defendants (Google and Character AI) argue yes. They claim that AI outputs, even those that harm children, are constitutionally protected speech. The judge rejected that argument—for now. But the fact that it was made at all is a warning signal we can’t ignore.

The Free Speech Shell Game

Larry Lessig has been sounding the alarm on this for years. In his 2021 essay “The First Amendment Does Not Protect Replicants,” he warned that machine-generated speech—produced without human intention or accountability—should not be considered protected speech. The First Amendment was designed to protect human expression in a democratic society, he argues, not the probabilistic outputs of code trained on scraped data.

“The replicant is not a person,” Lessig wrote, “it does not deliberate. It does not reflect. It just generates.”

Yet tech companies are advancing legal arguments that conflate human speech with machine output. Some go even further: claiming that if people want to hear the chatbot’s speech, then the speech must be protected, regardless of its impact.

As Jain puts it in this episode: “They’re not saying the chatbot has rights. They’re saying you have a right to hear the chatbot. It’s a back door—and it leads to complete immunity.”

When 18th-Century Law Meets 21st-Century Tech

One of the most sobering takeaways from the episode is how little legal infrastructure we actually have to address this moment. Courts are being forced to govern 21st-century technologies with 18th-century tools. There are no new federal laws in play. There are no expert regulatory bodies setting standards. And so the task of oversight falls to judges. Judges who are often under-informed, under-resourced, and courted by industry-backed lobbyists.

As Jain noted, this is “governance by litigation after the train wreck.”

The result is what CHT Policy Director Camille Carlton describes as a “snowball of precedent”—older rulings applied to technologies their authors could never have imagined. These cases build on one another, compounding their relevance until they quietly set the rules for the digital world. That’s how we got Section 230. That’s how we got Citizens United. That’s how we’ll get the next 50 years of AI law, unless we interrupt that momentum now.

The Slippery Slope to Personhood

If AI outputs are granted free speech protections today, what comes next?

Lessig and Jain both raised the specter of full legal personhood for machines. We’ve seen this pattern before: corporations granted limited rights to operate, then more rights, then political rights.

And with that leap comes legal protection for property ownership, campaign donations, contract enforcement, and even immunity from civil liability. That’s the endgame: systems more powerful than us, trained on our data, optimized to outcompete us in persuasion, and legally protected from consequence.

So What Do We Do?

Meetali Jain emphasized the importance of broad civic engagement. Courts can’t do this alone. Megan Garcia, in the wake of her unimaginable loss, has launched a foundation to help other families understand the risks. But this needs to be a national conversation.

Larry Lessig calls for a constitutional distinction between human speech and replicant speech. Not all code is speech. Not all outputs are opinions. We need to reassert the values behind the First Amendment—not stretch them beyond recognition.

Finally, regulation has to catch up. We need expert bodies that understand how these systems work. They need expertise. That means legal reform, new duties of care, and rethinking liability in the AI era.

This case isn’t about banning AI chatbots. It’s about whether we allow AI to operate beyond the reach of human responsibility.

Donate

In a recent episode of Your Undivided Attention, Daniel Barcay and Tristan Harris spoke with AI researcher Daniel Kokotajlo about his speculative forecast AI 2027: a detailed scenario depicting how competitive pressures could drive us toward dangerous superintelligence in the next two years, much faster than we're prepared to handle.

Kokotajlo, a former OpenAI researcher who left the company (and risked millions in stock options) to speak freely about AI risks, offers a sobering analysis of where our current trajectory might lead. The outcomes he predicts are scary—one path ends in human extermination—but the scenario isn't designed to scare, it's designed to clarify the competitive pressures pushing us toward potentially catastrophic outcomes so we can choose a different path.

The incentives behind the forecast…

The AI 2027 scenario is built on three key competitive pressures that reinforce each other:

• Corporate competition: Companies racing to beat each other economically, leading to faster development and deployment of AI systems without adequate safety testing.

• Geopolitical competition: Nations racing to ensure dominance in AI, creating pressure to move quickly regardless of risks. In his forecast, Kokotajlo predicts that "in early 2027, the CCP steals the AI from Open Brain so that they can have it too, so they can use it to accelerate their own research."

• The alignment problem: As companies rush to deploy increasingly powerful AI systems, they rely on training methods that don't reliably instill positive values, leading to AIs that appear aligned but are actually pursuing different goals.

…and the assumptions

AI 2027 is also built on some key assumptions that may not hold up:

• Scaling laws hold: The scenario assumes current scaling trends will continue unabated and breakthrough discoveries will happen on schedule. It also assumes that current architectural approaches will continue to work as systems become more powerful, without encountering fundamental technical barriers that could slow progress.

As Daniel Barcay notes: "AI timelines are incredibly uncertain, and the pace of AI 2027 as a scenario is one of the more aggressive predictions that we've seen."

• Institutions remain passive: The scenario assumes that democratic institutions will remain largely unable to meaningfully regulate or slow the pace of AI development. It doesn't deeply explore potential circuit breakers—moments where public pressure, technical setbacks, or catastrophic near-misses might force a slowdown or enable international cooperation.

• Misalignment is a given: The forecast assumes that alignment challenges will remain unsolved and that AI systems will become deceptive at scale. While there's already evidence that current AI systems can engage in deception when it serves their training objectives, the scenario assumes this capability will scale dramatically without corresponding advances in our ability to detect or prevent it.

An invisible acceleration

Most of the cutting edge AI research happens behind closed doors and under intense competitive pressure, meaning that sea-changes can happen quickly and without time for society to prepare.

As Daniel Barcay puts it: "It's pretty insane that for technology moving this quickly, only the people inside of these labs really understand what's happening until day one of a product release where it suddenly impacts a billion people."

This creates a massive information asymmetry where critical decisions about humanity's future are being made by a small group of corporate actors without meaningful public input or oversight.

Recursive self-improvement is critical

The main driver of Kokotajlo’s forecast is recursive self-improvement: the development of autonomous coding agents that can do AI research and development better than humans.

The trajectory outlined in AI 2027 shows how this might unfold: AI systems progress from basic coding assistance in 2025 to fully autonomous researchers in early 2026 who can "automate all the research" by mid-2027. At that point you have "something like a hundred thousand virtual AI employees that are all networked together, running experiments, sharing results with each other."

From there, it’s just a short burst to extraordinarily capable AIs. As Daniel puts it: "once you have AIs that are fully autonomous goal-directed agents that can substitute for human programmers very well, you have about a year until you have superintelligence, if you go as fast as possible."

Subscribe now

What do we mean by Superintelligence?

When AI researchers talk about superintelligence, they're not referring to a slightly smarter chatbot or a better chess-playing program. They're describing AI systems that surpass human intelligence across virtually all domains—from scientific research and engineering to strategic planning and creative problem-solving.

"OpenAI, Anthropic, and to some extent Google are explicitly trying to build superintelligence to transform the world," Kokotajlo explains. But the transformation they're envisioning goes far beyond automating routine tasks. These systems would be capable of conducting independent research, making breakthrough discoveries, and designing new technologies at speeds that dwarf human capabilities.

The key insight is that superintelligence represents a phase transition, not just an incremental improvement. Once AI systems become capable of improving themselves and designing their successors, the pace of change could accelerate beyond human comprehension or control.

This isn't science fiction speculation—it's what the leading AI companies are actively working toward, even as many of their own researchers acknowledge the existential risks involved.

The alignment problem

Core to the AI 2027 forecast is the assumption that AIs are fundamentally misaligned, that they will pursue goals that run counter to what’s best for human flourishing. It’s the kind of thing that you might see in science fiction but unlike science fiction scenarios where humans directly program goals into AIs, our reality is more precarious:

"They're giant neural nets. There is no sort of goal slot inside them that we can access and look and see what is their goal," Kokotajlo explains. Instead, we train these systems in environments and hope they develop the values we want—a process that’s unreliable and increasingly difficult to verify as systems become more sophisticated.

The scenario assumes that as AI systems become more sophisticated, they'll get better at hiding their true motivations—what researchers call "alignment faking."

"The AIs are often saying things that are not just false, but that they know are false and that they know were not what they were supposed to say," he notes. As these systems become more capable, they may become better at hiding their true motivations until it's too late to course-correct.

We’re already seeing some examples of this emergent misalignment when these models are red-teamed. Researchers have gotten these models to deceive their users, cheat at chess, threaten to download themselves onto external servers, and even blackmail engineers to avoid being shutdown.

Geopolitical Pressures: The US-China Dynamic

The AI 2027 scenario places geopolitical competition at the center of the race toward superintelligence. It depicts a world where national security concerns override safety considerations.

In the forecast, when China steals AI technology from US companies, it "causes a sort of soft nationalization/increased level of cooperation between the US government and Open Brain," Kokotajlo notes. This creates a feedback loop where each side's defensive moves accelerate the race.

The geopolitical dimension makes the coordination problem exponentially harder. Even if US companies wanted to slow down for safety reasons, the threat of Chinese competition provides a powerful justification for maintaining breakneck pace.

But the scenario also hints at the fundamental absurdity of this competition. Both sides are racing toward a technology that their own experts say could pose existential risks. It's a classic security dilemma where each side's attempts to ensure its safety through technological dominance actually increases the danger for everyone.

The international dimension also complicates any potential solutions. Transparency requirements, safety standards, and development moratoria become much harder to implement when they're viewed through the lens of national competitiveness. How do you convince a nation to handicap itself in what's perceived as the ultimate strategic competition?

As the scenario suggests, this dynamic could lead to a world where "citizens everywhere may not have a meaningful chance to push back" because the decisions are being driven by geopolitical imperatives that override democratic input.

What can we do now?

While the scenario is alarming, it's not inevitable. Kokotajlo emphasizes three immediate priorities:

• Transparency Requirements: Companies should be required to disclose their AI systems' capabilities, safety assessments, and development timelines. The public deserves to understand what's being built in their name.

• Whistleblower Protections: "One of the only enforcement mechanisms we have is employees speaking out basically," Kokotajlo emphasizes. We need legal protections for those with inside knowledge to speak up about safety concerns without sacrificing their livelihoods.

• Technical Oversight: "We need technical experts in alignment research to actually make those calls, and there are very few sets of people in the world, and most of them are not at these companies," Kokotajlo warns. Independent experts need protected channels to evaluate safety claims.

The stakes

The AI 2027 scenario forces us to confront an uncomfortable reality: the competitive pressures behind AI are pushing in a very bad direction. Whether the specific timeline proves accurate is less important than understanding how current incentives could lead us to lose control of or to our most powerful technology.

The question isn't whether AI will transform our world—it's whether we'll consciously shape that transformation or let bad incentives drive us toward outcomes nobody actually wants. The window for meaningful intervention is still open, but it may not remain so for long.

As Kokotajlo notes, the companies building these systems have stated that AI could pose existential risks, yet they continue racing toward superintelligence”

"We've got these important facts that people need to understand. These people are building superintelligence... many of the researchers at these companies, and then hundreds of academics and so forth in AI have all signed a statement saying this could kill everyone."

The bottom line

We stand at a crossroads where clarity about our current trajectory is essential for choosing a different path. The competitive dynamics driving AI development are real and powerful—but they're not inevitable. With transparency, oversight, and democratic participation in these decisions, we still have the power to steer toward a future that serves humanity rather than replacing it.

Donate to support our work

Recommended Media

The AI 2027 forecast from the AI Futures Project

Daniel’s original AI 2026 blog post

Further reading on Daniel’s departure from OpenAI

Anthropic recently released a survey of all the recent emergent misalignment research

Our statement in support of Sen. Grassley’s AI Whistleblower bill