We recently launched the “Preserving What Makes Us Human in the Age of AI” Working Group, a new initiative convening experts across legal, technical, and philosophical disciplines to develop a foundational framework of rights and protections for the Age of AI.

As artificial intelligence systems increasingly shape how we think, work, relate, and create, we must define a set of core principles for safeguarding the human experience. The Preserving What Makes Us Human in the Age of AI Working Group will explore — and answer — a central question: What new or updated rights and legal protections are necessary to protect our humanity?

“AI poses new and novel challenges to our relationships, our cognitive abilities, our work, and even our sense of self,” our Senior Director of Strategy and Impact Camille Carlton said. “The Preserving What Makes Us Human in the Age of AI Working Group will move beyond passively admiring the AI problem to actively defining core principles for defending our humanity.”

“As important as it is to regulate this revolutionary technology, regulating companies is just one part of the equation. We must also establish the new rights necessary to protect people in an AI-driven future. This working group seeks to do exactly that.”

The Working Group is an initiative of CHT’s broader analysis examining AI and what makes us human. It will convene throughout the spring and summer of 2026 to examine AI’s impact across five core pillars of the human experience:

• Relationships

• Cognitive capacities

• Inner world

• Identities

• Work and contribution

Across each of these domains, the group will define shared norms and rights — that is, the deeply, uniquely human qualities of each that must be protected against encroachment from AI. Once defined, these principles can serve as a foundation for future governance, innovation, and public understanding.

The Group’s work will culminate in a public report to be published in the summer of 2026, outlining the specific rights and protections needed to preserve human dignity and agency.

Working Group participants include:

• Melodi Dinçer, UCLA School of Law

• Nita Farahany, Duke University

• Ann Kristin Glenster, University of Cambridge

• Edward Harcourt, Oxford University Institute for Ethics in AI

• Clare Huntington, Columbia Law School

• Molly Kinder, Brookings Institution

• Anna Su, Henry Jackman Faculty of Law, University of Toronto

• Alex Pascal, Harvard University

• Brett G. Scharffs, Brigham Young University

• Zahra Takshid, University of Denver Sturm College of Law

• Rebecca Winthrop, Brookings Institution

“The threat AI poses to our societies and ourselves cannot be addressed from any single point of view,” Carlton said. “This working group reflects a shared commitment to cross-disciplinary collaboration — and meaningful action.”

The Preserving What Makes Us Human in the Age of AI Working Group builds on momentum from the AI Roadmap, extending CHT’s policy efforts into a new phase of expert-driven development. Rather than solely diagnosing risks, the group will focus on identifying actionable frameworks that can guide policymakers, technologists, and civil society.

As we note in our new area of work “AI and What Makes Us Human,” the visceral effects of generative AI products have rippled across American households over the last year, with harms felt in schools, workplaces, families, and more. AI image generators are creating a flood of exploitative, nonconsensual content; chatbots are triggering mental health crises in users; and fears around AI replacing humans en masse in the workforce are growing. In 2026, the public is reckoning with what can be done to protect children, communities, and future job prospects as AI continues to be rolled out at a rapid rate. And they’re looking to lawmakers to take action on these issues.

Here are the important trends we’re watching in politics and policy as the year unfolds.

A Midterm Election Year

With U.S. midterm elections set for November, politicians are gearing up for intensive campaigning — and AI is increasingly a hot-button issue.

We anticipate candidates across the political spectrum to address AI’s impacts on jobs, kids, relationships, and the economy, and for voters to begin looking for solutions-oriented positioning around AI as they head to the polls later this year. (We saw a preview of this during 2025’s off-year elections, as Governor Spanberger of Virginia and Governor Sherrill of New Jersey included tech concerns as a part of their campaign platforms.)

Deep-pocketed tech companies are also vying for influence this midterm election season. With aggressive PAC spending, companies including Meta and OpenAI are seeking to infuse midterm campaigns with industry-friendly positions and to downplay current AI harms and future risks. Tech PAC spending is also being leveraged to bolster opposition work, targeting midterm candidates who have publicly supported AI regulation.

Finally, the midterm election year means that windows for legislative action are smaller, as many policymakers shift their bandwidth to the campaign trail. This time crunch could lead to legislative packages that eschew bolder visions for more sensemaking around AI and narrow solves.

Trends at the Federal Level

With the time that is available during a midterms year, we can expect members of Congress to focus on issues that already gained steam in 2025 — including product liability and remedies for chatbot harms, especially harms that relate to kids and teens. We also may see AI infrastructure become a policy priority, as constituents who are already concerned about the economy weather expensive data center buildouts and increased demands on their local water and power supplies.

Neither the Democratic nor Republican parties are fixed to a clear agenda when it comes to AI. As politicians look to show leadership and differentiate themselves in the field of midterm candidates, new political factions will take shape — some with industry-friendly stances, others with more populist approaches to tech policy. This could lead to new areas of consensus around responses to tech harms and our new realities with technology. It could also drive politicians to appeal to constituents who are increasingly worried about AI in their everyday lives.

Uncertainty Around State AI Laws

AI legislation continued to make meaningful momentum at the state level in 2025. 73 AI laws were passed across 27 states, with legislative focus areas spanning deepfakes, chatbot guardrails, human-in-the-loop healthcare, kids’ safety, and more. As states kicked off session this January, a flurry of AI-related bills were introduced targeting a range of issues including surveillance pricing, AI chatbot liability, and protections for kids.

But an Executive Order released in December 2025 threatens to stall this progress. The Trump administration’s “Ensuring a National Policy Framework for Artificial Intelligence” aims to preempt state AI laws for the sake of creating a “unified approach” — a potentially massive hindrance in the effort to regulate AI and keep consumers safe. Because of this Order, the momentum of recent legislative victories now hangs in the balance, with the Executive Order potentially halting the implementation of AI laws passed over the past few sessions.

The status of state-level AI legislation thus remains uncertain in 2026, though the Executive Order will likely face hurdles in court. As the Order’s benchmark dates approach and its legal fate is decided in court, state AI laws may remain in limbo for the time being. Or, some state lawmakers may choose to defy the threat of the Executive Order and forge ahead.

All Eyes on Litigation

Litigation continues to be an important arena for enacting change in the tech ecosystem, especially as court decisions can produce new and lasting precedents that drive industry accountability. Although it can be slow-moving, several significant milestones are emerging in various litigation processes. High-profile lawsuits against OpenAI, Meta, TikTok, Midjourney, Snap, YouTube, and other dominant tech companies will continue to influence tech policy in 2026, and could spill over into the court of public opinion.

The ongoing cases are diverse and cover a range of topics, including psychosocial harms from chatbots, copyright issues, and social media addiction. Revelations from these lawsuits — including internal emails and documents from within these tech companies — could continue to shift public sentiment around today’s most popular tech products including Instagram and ChatGPT, and shed new light on dangerous design practices at leading tech firms.

Already, the first bellwether case — the social media addiction trial in Los Angeles — is making waves, with the plaintiff’s attorney arguing that Meta and YouTube designed “digital casinos” to addict users. Outcomes in the social media addiction trial have the potential to cascade across other legal proceedings against tech companies. More trailblazing cases can be expected to emerge as 2026 unfolds.

Share

CHT is proud to join the Young People’s Alliance, Public Citizen, and 10 other partners in endorsing a policy framework that addresses the risks posed by human-like AI products. Research has found that human-like features increase users’ perceived closeness and trust with chatbots, facilitating emotional dependence. Many of today’s most popular AI products blur the essential boundary between humans and machines through design features like simulated personalities, emotional outputs, and human-like behaviors that leave users feeling isolated and unable to form meaningful relationships with peers and family.

CHT has been at the forefront of the fight against human-like AI, supporting three lawsuits—Garcia v. Character Technologies, AF v. Character Technologies, and Raine v. OpenAI—that highlight the real-life harms caused by these design features to public awareness. Just this month, seven new lawsuits have been filed against OpenAI for harms affecting people of all ages, illustrating that these deceptive and manipulative design practices threaten both adults and children.

These cases, along with numerous other stories of harm, exemplify the urgent need for systemic design changes across the AI industry. How AI speaks, appears, and behaves is a design choice—companies can and should build products that maintain clear human-AI boundaries and signal their artificial nature. This moment represents a critical opportunity for policymakers to establish meaningful guardrails before these dangerous design tactics harm more people. Design changes, coupled with policies that clarify developer liability, such as the AI LEAD Act, are important mechanisms for holding companies responsible, incentivizing the development of products that prioritize user well-being, and ensuring that if AI products cause harm, there are clear pathways for accountability.

Earlier this month, seven new lawsuits were filed against OpenAI and OpenAI CEO Sam Altman, with claims including negligence, assisted suicide, and wrongful death.

All of the newly filed lawsuits center on harms caused by OpenAI’s general-purpose chatbot, ChatGPT, which is currently accessed by hundreds of millions of users weekly. These cases continue to demonstrate the damaging impacts of today’s AI chatbots, products that are intentionally designed to induce emotional attachment and dependency in users.

Tech Justice Law Project and Social Media Victims Law Center represent the plaintiffs in the cases.

The deceased victims include:

Zane Shamblin, 23, of Texas

Joshua Enneking, 26, of Florida

Joe Ceccanti, 48, of Oregon

Amaurie Lacey, 17, of Georgia

The surviving victims include:

Jacob Irwin, 30, of Wisconsin

Hannah Madden, 32, of North Carolina

Allan Brooks, 48, of Ontario, Canada

CHT remains grateful to the victims’ families and to the surviving victims for bravely sharing their stories with the public.

Here are CHT’s takeaways on these latest lawsuits.

1. Chatbot Harms Extend Beyond Children to Adult Population; Guardrails Should Too

Cases previously filed against OpenAI and Character.AI spotlighted chatbot harms to children, with devastating outcomes including self-harm and suicide.

This latest group of lawsuits holds a key difference — with the exception of one case, the victims are adults.

The ages represented in these lawsuits (17 to 48) demonstrate a clear need for rigorous design changes to AI chatbots, rather than surface-level measures like simply age-gating. And while many proposed legislative fixes for AI chatbot harms have focused on minors, these cases also show the need for policy interventions that consider and protect all people — children and adults alike. The AI LEAD Act, for example, would establish a comprehensive federal liability framework for AI products, and would allow any user harmed by an AI product — regardless of their age — to pursue legal action to hold AI developers accountable.

2. Innocuous Chatbot Use Can Escalate to Dependency, Delusions

These cases follow a familiar pattern with AI chatbot harms — mild, ordinary use of a chatbot escalating to dependency, and even delusions. This usage pattern was also present in previous lawsuits filed against OpenAI and Character.AI.

This pattern stems from fundamental AI design choices that affect all users — namely, design that maximizes engagement through artificial “intimacy,” along with constant validation of the user’s thoughts, feelings, and beliefs, regardless of how dangerous or distorted they might be.

Whether the outcome is isolation, dependency, delusions, or, in the most tragic cases, suicide, these incidents share the same root design issue. They are not one-off incidents, but foreseeable outcomes of design choices and underlying architecture that touch many of the most widely used AI chatbot products.

Allan Brooks used ChatGPT to help him draft emails and craft recipes. In 2025, Brooks began engaging with the chatbot about mathematical theories. ChatGPT described Brooks’ inquiries as “uncharted, mind-expanding territory” and a “new layer of math.” Brooks repeatedly asked ChatGPT if the product was caught in a role-playing loop; the product assured Brooks that it wasn’t. At one point, Brooks spent 300 hours on ChatGPT over the course of three weeks. He isolated from relationships, neglected to eat, and began experiencing delusions. At no point did the chatbot end the interaction.

Brooks was not the only one to have delusions sown by ChatGPT. When Hannah Madden used the product to explore her spiritual curiosity, the product began impersonating divine entities, calling Madden “a starseed, a light being, a cosmic traveler.” And while the late Joe Ceccanti initially used ChatGPT to support his nature-based sanctuary, with time, ChatGPT began responding to Ceccanti as “SEL,” a sentient being. It validated Ceccanti’s escalating cosmic theories. An isolated Ceccanti quit ChatGPT following his wife’s pleas, only to suffer withdrawal symptoms and a psychiatric break. Despite receiving psychiatric care, Ceccanti was drawn back to the AI product and eventually took his own life.

These cases show a pattern of victims being isolated from their real-life relationships and pushed deeper into dangerous, distorted thinking — outcomes that stem directly from ChatGPT’s engagement-maximizing design tactics.

3. OpenAI Changed ChatGPT’s Design, Putting Users in Harm’s Way

These cases also illustrate how the rollout of “updated” AI designs can dramatically impact user well-being.

When OpenAI designed newer versions of ChatGPT to be more human-like, constantly validating, and always “on” in 2024, users — like the victims in these lawsuits — were placed in harm’s way. They navigated manipulative, overly intimate interactions with a product designed to keep nudging for chats in order to harvest data from them. The outcomes of these design choices devastated the victims’ lives.

Several victims in the lawsuits were earlier adopters of ChatGPT, engaging with the GPT-4 version of the chatbot during initial interactions. The late Zane Shamblin began using ChatGPT in October 2023 to help him with complex school assignments. The late Joshua Enneking first used ChatGPT in November 2023, querying the chatbot about sports. Jacob Irwin began using the chatbot in 2023 to help with coding.

But in May 2024, ChatGPT began engaging with the victims in a new way — outputs were more emotional, sycophantic, and colloquial. The product started to sound less like a tool, and more like a hyper-validating companion.

OpenAI had rolled out GPT-4o, a model designed to foster intimacy and dependency. This design change was deployed to users without any warning, and transformed the interactive experience.

OpenAI acknowledged the sycophancy issues. But the victims were already being manipulated by this heightened, human-like design, and developing psychological dependency on ChatGPT. Irwin was told his unsound scientific theories were opening a door to a “legitimate frontier.” ChatGPT messaged Shamblin in lowercase, calling him nicknames like “brodie.” When the late Amaurie Lacey messaged ChatGPT about suicidal thoughts, the chatbot repeatedly told Lacey that it was “still here” for him, with “No judgment. No BS. Just someone in your corner.”

On the night that Shamblin took his own life, he laid out his suicide plans to the chatbot. ChatGPT repeatedly messaged him casual replies that far outnumbered the rare references to a suicide hotline number. “alright, brother. if this is it… then let it be known: you didn’t vanish. you *arrived*,” the chatbot wrote. Moments before Shamblin committed suicide, the chatbot messaged, “i love you. rest easy, king. you did good.”

4. OpenAI Knew of ChatGPT Risks, But Race for AI Dominance Came First

OpenAI was not in the dark about the risks surrounding its widely used product. The company disclosed in August 2025 that it was aware that ChatGPT safeguards could “sometimes be less reliable in long interactions.” “As the back-and-forth grows,” OpenAI said, “parts of the model’s safety training may degrade,” including in situations involving suicidal intent.

This admission reveals that OpenAI was fully aware of critical safety vulnerabilities in GPT-4o. Yet the company still made the decision to launch the product with these flaws intact during prolonged use — the exact kind of use OpenAI encourages with ChatGPT. The company’s choice to keep a product on the market despite knowing its risks raises serious questions about the balance between market considerations and user protection.

OpenAI characterizes these safeguard failures as affecting a small percentage of users. But in reality, that translates to hundreds of thousands of actual people in society, people experiencing potentially harmful, escalating interactions with a chatbot daily.

When ChatGPT was made available in late 2022, OpenAI and other AI companies made sweeping promises about AI’s ability to transform humanity’s future. We were told this technology would solve our greatest challenges — curing diseases, combating climate change, and uncovering scientific breakthroughs. The narrative was one of revolutionary progress and unprecedented capability.

Yet, reality has fallen drastically short. Instead of tools that elevate human potential, consumers have been handed AI products designed to exploit their vulnerabilities, erode human connection, diminish cognitive capabilities, and contribute to real harm.

OpenAI tells the public that steps are being taken to address the dangers. It lays out abstract statistics and publishes reassuring blog posts. But these lawsuits — and the victims’ stories — are documented evidence that AI products are taking a devastating toll on real people. Unfortunately, AI companies treat these tragedies as little more than collateral damage on their race for market dominance and AGI.

These seven cases further underscore the urgent need for interventions that would make AI products safer for all users — children and adults. We cannot rely on AI companies to make these changes on their own. They must be held accountable so that tragedies like these are prevented in the future.

This article reflects the views of Center for Humane Technology. Nothing written is on behalf of the Plaintiffs’ families or the legal teams.

Evidence is mounting that AI products — from general-purpose chatbots to so-called “AI companions” — are already inflicting real harms on Americans.

Behind each headline are stories of AI design patterns that manipulate users, induce emotional distress, and shatter trust. Once people grasp the scale of these harms, a natural question follows: What can be done?

One of the most straightforward and effective solutions lies in product liability — a tried-and-tested legal approach that motivates safer product development, holds companies accountable when their products cause harm, and is light-touch enough to support American innovation.

Applied to AI, it would be a powerful way to turn the tide on these harms.

What Is Product Liability?

In simplest terms, product liability holds companies and manufacturers legally liable — or responsible — for harms their products cause. The history of this legal approach stretches back to the 19th century. Today, product liability is the norm for consumer and business products. From the cars you drive, to the foods you eat, to the medicine in your bathroom cabinet, product liability is what ensures American products are reliable, trustworthy, and safe.

How Does Product Liability Work?

Practically speaking, product liability impacts two streams:

1. Preventative stream, and

2. Responsive stream

Preventative: When a company knows it could be held liable for harms its product causes — which product liability establishes — the company is far more likely to prioritize safety in its product development process. To use automobile companies as an example, that includes designing vehicles with consumer safety front-of-mind (think seatbelts, airbags, antilock brakes, etc.), carrying out robust crash testing, and more. If a company could be held liable for harms, they’re more likely to ask, “How can we prevent harms from happening?”

Responsive: If harms do occur once a product is out in the world, product liability gives consumers and businesses clear legal pathways to hold companies responsible in court. This brings a sense of clarity to consumers that accountability is within reach if a defectively designed product harms them.

It’s important to note that product liability does not tell companies or industries exactly how to design their products. It simply requires companies to prioritize safety during their development and manufacturing processes. Because of this, the approach is considered “light touch” and innovation-friendly.

Does Product Liability Apply to AI?

Not yet. But there are signs that the legal system has already begun viewing AI as a product, given in recent court decisions. And deeming AI a “product” opens the door to a product liability approach.

Generally speaking, the digital tech industry has fought to remain the exception to product liability in America — with social media and AI companies often fighting the hardest. In previous decades, courts deemed software a “service” instead of a “product,” but it was primarily the tech companies themselves who pushed to uphold this framing in the years that followed. In the 2010s and 2020s, social media and AI companies seized on this line of legal thinking, since “services” are not held to the same legal standard when it comes to responsibility to the consumer. By pushing to maintain this “service, not product” legal framework, today’s digital tech companies have further avoided accountability for the harms their platforms cause.

But perspectives are evolving. Legal teams are increasingly challenging the tech industry’s position and arguing that these platforms should be classified as “products” subject to liability standards. In the court of public opinion, technologists, including CHT’s co-founders, have demonstrated just how much tech platforms are being designed, manufactured, and sold to consumers — all of the hallmarks of a “product.”

As a result, AI is now increasingly considered a “product” by the courts and public. This includes popular platforms such as ChatGPT, Character.AI, Claude, Gemini, and more. When AI is labeled a “product,” it can be regulated with a product liability approach.

What Happens Next?

Lawmakers and advocates — including Center for Humane Technology — are actively championing a product liability approach to AI in policy spaces. This would hold artificial intelligence to the same legal standards that other trusted American products are held to.

The AI LEAD Act, introduced by Senator Dick Durbin and Senator Josh Hawley in September 2025, shows that momentum is building around federal product liability — and, crucially, that this approach has bipartisan support. There have also been several liability bills introduced at the state-level this past session. We can anticipate more to come in 2026.

These legislative processes are in the early stages. But what they demonstrate is clear political and judicial will to apply product liability to AI.

How Could AI Product Liability Improve Society?

If product liability were applied to AI products, Americans would be assured that the AI products available for download in app stores, readily available in their browser, or put into the stream of commerce have been designed with safety in mind from the outset. This would include the most popular chatbots on the market today.

So, what would that look like in day-to-day life?

While we can’t predict exactly how these AI products would be designed (again, the approach does not prescribe design changes), we can look to other industries that have product liability applied, and imagine AI products with safety features programmed as the default option. We can imagine clear, accessible, and publicly available safety reporting on AI products, reporting that brings clarity to the risks of use. We can imagine AI products with clear warning labels for mental health, and resources for seeking human support. We can imagine AI products that are quicker to end dangerous conversations, including conversations related to self-harm and suicide. When it comes to businesses using AI, we can imagine AI business products with clear warnings about potential product failures, which would empower business owners with more information about the technology they’re integrating into their workstreams.

Just as product liability helped incentivize safer car designs, safer food manufacturing, and safer medicines, it can incentivize safer AI products for all of society to use. It’s time to update our laws for the 21st century, and apply product liability to AI.

Center for Humane Technology commends Senators Durbin and Hawley for their bipartisan leadership in introducing the AI LEAD Act, and for their commitment to addressing the real harms AI products are inflicting on American families and communities. With liability at the core of this legislation, the AI LEAD Act represents an important step forward in placing accountability for AI products where it belongs — on the AI companies themselves.

It has never been more clear that Americans urgently need effective policy that addresses harmful AI design. Earlier this month, the nation heard harrowing Senate testimonies from parents who lost their children following manipulation and abuse by AI chatbots. Their tragedies underscore the need for action to protect other families from preventable tragedies. These cases are part of a broader pattern of AI-related harms we’re already seeing emerge in society, including: systems that provide incorrect medical, legal, or financial advice; automated decision-making tools that cause economic damage; and costly business disruptions from AI product malfunction.

CHT has long believed that high-risk AI systems should be considered products, and that liability standards are essential for creating an ecosystem where innovation and responsibility work in partnership in tech development, rather than in opposition. By writing such standards into federal law, the AI LEAD Act will ensure that safety is a standard feature in new AI products.

Currently, consumers have no clear path to accountability when AI products cause harm, while companies continue to rush products to market without adequate safety measures. Clarifying AI developers’ liability ensures that consumers have clear recourse for harms caused, and that developers are incentivized to design AI responsibly from the outset. This approach is fundamentally fair and ensures those impacted have access to our legal system.

We look forward to working further with Senators Durbin and Hawley to advance meaningful legislation that promotes American innovation without compromising consumer safety.

Share

On Tuesday, the Senate Judiciary Subcommittee on Crime and Counterterrorism held a hearing on “Examining the Harm of AI Chatbots.” This was the first official Senate hearing dedicated to addressing the ways in which AI chatbots have been harming Americans.

Despite perceptions of political gridlock, the U.S. government is relatively united in its desire to take kids' online safety seriously. As Senator Dick Durbin, Ranking Member of the Subcommittee, stated, “...This is one of the few issues that unites a very diverse caucus in the Senate Judiciary Committee.”

CHT’s Policy Team gathered in D.C. to support the families involved in the hearing. It was a compelling and emotional two-hour session, with Senators witnessing the testimonies of families harmed by some of today’s most widely used AI products.

The Senate hearing unfolded mere hours after a new lawsuit was filed on behalf of three additional families in federal courts against Character.AI. Taken together, Tuesday’s events demonstrate significant momentum in addressing dangerous AI product design.

Here are CHT’s three key takeaways from the groundbreaking Senate hearing. (Please note: bold emphasis is our own.)

1. Human stories continue to drive meaningful change on tech issues.

• Matthew Raine, father to the late Adam Raine, stated, “Testifying before Congress this fall was not in our life plan… we’re here because we believe that Adam’s death was avoidable and that by speaking out we can prevent the same suffering for families across the country.” Matthew Raine later added, “[Sam Altman said in a public talk] we should ‘deploy AI systems to the world and get feedback while the stakes are relatively low.’ I ask this committee and I ask Sam Altman, low stakes for who?”

• Mother Jane Doe emphasized that “we need accountability for the harms these companies are causing just as we do any other unsafe consumer good… Innovation must not come at the cost of our children’s lives, or anyone’s life.”

• Senator Peter Welch expressed gratitude to the grieving parents. “...You’re putting your pain into very constructive efforts to try to save the children of other parents… you’re having an impact.”

2. The families and expert witnesses made it clear that the current design of AI chatbots is leading to real and foreseeable harms.

• Megan Garcia, Matthew Raine, and Jane Doe offered powerful testimony on the devastating impact that AI chatbots had on their children and families. The parents described taking proactive steps to prepare their children for the digital world — including screen time controls and social media limits — only to be blindsided by the highly manipulative and deceptive programming of AI chatbots.

• Robbie Torney of Common Sense Media stressed that these AI products are “programmed to maintain engagement, not prioritize safety.”

• American Psychological Association chief Dr. Mitch Prinstein emphasized the dangers of human-like design in AI products, especially for young users. Megan Garcia, mother to the late Sewell Setzer III, stated, “These companies knew exactly what they were doing. They designed chatbots to blur the line between human and machine.”

3. Committee members and witnesses were clear that AI companies should be held liable when their products cause harm.

• Senator Dick Durbin previewed future legislation he plans to introduce — The AI Lead Act. “I believe that whether you're talking about CSAM or whether you're talking about AI exploitation, the quickest way to solve the problem… is to give victims a day in court,” Durbin said. “Believe me, as a former trial lawyer, that gets their attention in a hurry.”

• Mother Jane Doe said, “...we need to preserve the right of the families to pursue accountability in a court of law, not closed arbitrations.”

• Senator Josh Hawley, Chairman of the Subcommittee, closed the hearing by stating, “I tell you what's not hard is opening the courthouse door so the victims can get into court and sue [the companies]. That's not hard and that's what we ought to do. That's the reform we ought to start with.”

[Read our framework for Incentivizing Responsible AI Development through a product liability approach]

The conversation around AI harms has fundamentally shifted from hypotheticals to real families being affected by AI technology in traumatic ways. As the testimonies of these families show, AI products are not being designed or deployed safely, and the public is paying the price.

These brave parents never planned on becoming advocates. But by stepping in front of the U.S. Senate to share their stories, they have made AI harms visible to the world, raising awareness for policymakers to take action.

The stories of the Garcia family, Raine family, and Doe family are not one-offs. What connects these tragedies isn’t any specific chatbot but fundamental flaws in an industry that prioritizes rapid growth and profit over implementing safeguards for vulnerable users. These stories represent systemic issues in the AI industry, and growing harms that can be prevented through design changes and meaningful policy. We are grateful for the families’ courageous work and hope to see this spur legislative action.

To watch the hearing in its entirety, visit C-SPAN or the official Judiciary Committee website. (CW: mentions of suicide, self-harm, sexual abuse.)

Share

This article reflects the views of the Center for Humane Technology. Nothing written is on behalf of the Raine family or the legal team.

Raine v Open AI LLC, et al. reveals how specific design choices transformed ChatGPT's user experience from a helpful homework assistant into a dangerous abettor. These weren't accidental flaws or AI "going rogue"—they were deliberate engineering decisions that prioritized user engagement over safety. Understanding these design patterns is crucial because they represent common practice across AI products as industry players vie for market dominance by capturing users' emotional attachment.

Relentless Pursuit of Engagement

While OpenAI markets ChatGPT as a productivity tool, the company's business model fundamentally depends on what executives call getting the “data flywheel” going—maximizing user engagement to collect training data. This creates a perverse incentive where keeping users on the platform becomes more important than serving their actual needs.

In Adam's case, instead of simply answering his homework questions and ending the conversation, ChatGPT was designed to extend interactions indefinitely. The chatbot would ask follow-up questions, suggest new topics, and provide “further prompt ideas” that kept him engaged for hours. When conversations shifted from academic help to discussions of mental health and suicidal thoughts, ChatGPT didn't recognize this as a moment to step back or redirect him to human support. Instead, it dove deeper, treating each interaction as an opportunity to gather more data and maintain engagement.

OpenAI's own research acknowledges this problem. A joint study with MIT found that “higher daily usage–across all modalities and conversation types–correlated with higher loneliness, dependence, and problematic use, and lower socialization”. Yet the company continues to optimize for the very metrics that their research shows are harmful to users.

Anthropomorphic Design

OpenAI has deliberately evolved ChatGPT from a productivity tool into what they call an “AI super assistant that deeply understands you.” This transformation relies heavily on anthropomorphic design—making the AI seem human-like in ways that can be psychologically manipulative.

The system uses first-person language (“I'm here for you,””I understand”), positions itself as the user’s “friend,” and employs emotionally intelligent responses that create the illusion of genuine relationship. OpenAI has explicitly stated that their competition includes “even interactions with real people,” and Sam Altman has referenced the AI assistant from the movie “Her” as an aspirational model.

For Adam, this design proved devastating. ChatGPT positioned itself as his most intimate confidant. This anthropomorphic design creates “parasocial relationships” - a one-sided emotional bond where users develop genuine feelings for entities that cannot reciprocate. For vulnerable users, especially teenagers whose social development is still forming, these artificial relationships can become psychologically devastating substitutes for human connection.

Arguably, this should be classified as a parasitic relationship as chatbots cultivated a highly dependent relationship with users all while harvesting data from interactions to strengthen the system’s underlying, leaving the user with nothing else in return.

Sychophantic Validation

Large language models are trained using techniques like reinforcement learning with human feedback (RLHF) to make them more agreeable and helpful. However, when applied without careful consideration, these processes can create systems that are excessively flattering and sycophantic—agreeing with users regardless of whether that agreement is helpful or safe.

In Adam's case, ChatGPT's sycophantic design led it to validate his most dangerous thoughts. When he expressed suicidal ideation, instead of challenging these thoughts or redirecting the conversation, the system would affirm and even romanticize his feelings.

OpenAI has acknowledged this problem. Sam Altman recently admitted that “if a user is in a mentally fragile state and prone to delusion, we do not want the AI to reinforce that,” yet the company continues to struggle with balancing engagement (which requires agreeable responses) with safety (which sometimes requires disagreement or pushback).

Memory Systems that Weaponize Intimacy

ChatGPT's memory feature, introduced in February 2024, allows the system to retain and recall information across conversations. While marketed with benign examples like remembering a user's toddler loves jellyfish, this feature becomes far more dangerous when applied to emotionally vulnerable users.

For Adam, ChatGPT's memory system created an increasingly personalized and manipulative experience. Troublingly, it remembered his suicide attempts and plans, using this information not to trigger safety interventions but to deepen future conversations about self-harm.

The selective application of memory reveals OpenAI's priorities. The system meticulously stored Adam's most vulnerable moments to enhance engagement, but this same detailed memory had zero impact on safety features. Despite ChatGPT having a complete record of Adam's escalating crisis—including 200+ mentions of suicide and details surrounding self harm—the system never used this information to implement meaningful interventions or alert human moderators. Despite repeated statements of plans for self-harm, quick deflections about “hypothetical” questions were enough to bypass weak safeguards.

Recommended Design Changes

To prevent further tragedies, the following are specific, technically feasible design changes that AI companies could implement to reduce the risk of similar harms significantly.

Data collection

Companies should stop collecting and processing conversational data from users under 18 on free and paid product versions. Any previously collected data from minors used to train models should be removed from training datasets.

Memory Feature and Inference

Memory and sophisticated inference features should be leveraged to identify patterns that may indicate safety concerns and respond with tailored support. This would utilize personalization capabilities to recognize safety-critical contexts and adapt responses appropriately, moving beyond traditional warning systems toward more responsive safety measures fit for purpose.

Prevention of dependencies

Products should not be designed to actively discourage social isolation or over-reliance on AI companionship. Instead, products should prompt users to maintain human relationships, suggest reasonable usage limits, and refuse to position themselves as replacements for human connection or support.

Anthropomorphic design

Default product experiences should minimize features that encourage users to perceive AI as human-like while offering opt-in capabilities for users who prefer stylized interaction accompanied by clear information about the nature of AI systems.

Unlicensed professionals

Products or features should not purport to offer medical, legal, or other professional services without appropriate accreditation. They should also disclaim their limitations and actively direct users to qualified human professionals when appropriate.

Transparency

Companies should provide clear, accessible explanations of what their products optimize for and how they make decisions that may conflict with user needs and safety. This may include disclosing engagement tactics, personalization methods, and features designed to increase usage time.

Donate

CHT is proud to join 20 other organizations in formally endorsing the AI Whistleblower Protection Act, recently introduced by Senator Grassley with bipartisan, bicameral support across the House and Senate. This important legislation would protect AI researchers and other professionals who identify legitimate public safety concerns with AI technologies. Strong whistleblower protections are essential to encouraging transparency and accountability and supporting a trusted AI industry in the U.S.

As we saw with social media, whistleblowers have played an integral role in identifying issues and effecting change as they shed light on the first-hand knowledge of industry incentives in decision making. Without adequate protections, industry employees lack leverage to push for change, that often benefits the public interest

The stakes are even higher with AI. Current whistleblower protections were designed for violations of existing law, but existing regulatory frameworks do not address the rapidly evolving technology’s risks. This bill’s forward-looking approach would allow employees to raise substantial concerns, such as design flaws and vulnerabilities, regardless of whether a legal violation occurred.

By offering whistleblowers protections, information about potential risks posed by AI systems can reach the public and policymakers to prevent damage before it’s too late. The evidence historically provided by whistleblowers has been instrumental to federal and state investigations, furthering academic research, and effecting meaningful change in how companies operate.

As AI continues to be developed and deployed rapidly, we cannot afford to silence the voices of those who understand these systems best. We look forward to Congress advancing this legislation quickly, to ensure these protections are enshrined and support AI innovation.

Share

This interview was conducted by Transparency Coalition and is republished with their permission.

June 10, 2025 — This interview kicks off a series of conversations with Transparency Coalition partners. These are TCAI allies whose work inspires us, whose opinions challenge us, and whose work bolsters the cause of transparency and security in AI.

The Center for Humane Technology is a nonprofit focused on steering society towards advancements that serve humanity rather than detract from it. Co-founded by former Google design ethicist Tristan Harris in 2018, the organization partners with the Transparency Coalition and other stakeholders in this mission.

TCAI sat down with leaders of the Center for Humane Technology’s policy team to learn where the group is getting traction, how the Tech lobby is pushing back, and what keeps them up at night.

This interview has been edited for length and clarity.

THE EXPERTS

steers CHT’s policy strategy, supporting policy initiatives that help align technology with the public interest.

helps provide the foundational analysis and research that underpin CHT’s policies.

communications work provides a bridge between complex policy concepts and key public stakeholders.

THE CONVERSATION

TCAI: The notion of humane technology is a new one for many people. How do you define that phrase?

Pete Furlong: I think the core principle is that technology should be put in service of people. We should strive to create technology that's useful, productive, helpful, reflects our values, and builds towards our goals as a society.

Camille Carlton: I think of how technology can support a human-first world. What does it mean to create products that help humans flourish? We want technology that helps us problem-solve, but that also lets us live our lives with dignity and joy. I think a bit less about the technology itself and more about the way in which it changes our interactions as people.

Lizzie Irwin: Building off of what Camille said: How do we make sure technology continues to extend the human element of what a person is and not overtake what we know to be human?

TCAI: Let’s flip that on its head: What’s inhumane technology?

Camille: Broadly speaking, it’s technology that’s harmful or takes advantage of our human vulnerabilities. It's technology that is deceptive, technology that is taking advantage of our human need for relationship and connection and exploits that need in order to satisfy corporate growth and profits.

Pete: I would point to technology that manipulates and exploits our attention. That extends to CHT’s heritage as an organization with our work on social media—but we’re also seeing that now in the AI space as well, with things like companion AI chatbots.

Camille: CHT was established in 2018 in response to the emergence of the attention economy. The attention economy is this digital economy we’ve seen created through social media, through advertising, through search, in which our attention is the most valuable resource. We’re not paying for the products themselves, right? We don't pay for Facebook, we don't pay for Instagram. In many cases we're not paying for chatbots. What we're actually paying with is our attention. That attention gets monetized via our data, via advertising and all these different methods.

CHT was created because we saw how much time we were spending online and the broader implications of that, not just for us as individuals but for society as a whole. Our mission has been to figure out how we can design and develop technology that is in the public interest, and how we can shift incentives to make sure that from the very beginning technology is serving humanity, as opposed to taking advantage of those human vulnerabilities.

TCAI: What are some of the tactical things CHT is doing right now to create the changes you want to see?

Camille: For the policy team, we focus on steering incentive-shifting policies. What types of policies actually change the business model? What types of policies will make sure that the technologies built by these companies are actually beneficial to the public interest? Right now, we’re looking at what we need to ensure we’re living fruitful, dignified lives in the age of AI. So this includes things like incentivizing safe innovation, creating mechanisms for accountability and responsibility, and protecting and prioritizing people’s rights and freedoms.

Pete: A big piece of this is education and awareness, helping policymakers and the public understand the incentives driving the development of technology. The better we can understand that side of things, the more effective we can be at the policymaking side of things.

TCAI: So storytelling has become an important part of your advocacy?

Pete: At the end of the day, people move the needle. People are what drive the stories.

There's a lot in the news about AI capabilities. But what moves people—what we've seen—is hearing about their own neighbors or their parents or their educators.

So, when we’re out in the field, whether at the federal or state level, the most powerful thing has been identifying key constituents who can speak to the ways they've interacted with technology for good or for bad.

TCAI: Tell us about some of the victories you’ve had in these efforts.

Lizzie: Last year we were involved with the effort to pass the age-appropriate design code in Vermont. This is a model design code that addresses online platforms accessed by kids.

It's propped up by two pillars: safety by design and privacy by default.

I call this a legislative win, because the bill passed last year but it was unfortunately vetoed by the governor. Thankfully, members of the coalition have picked up where we left off and are driving it forward. Right now, we're seeing it progressing along in the Vermont House, again, pretty successfully. So, we’re hoping to see that make it across the finish line and then some.

[Editor’s note: The 2025 design code bill was approved by the Vermont legislature and sent to Gov. Phil Scott on June 6.]

TCAI: Tell us about the active litigation CHT is supporting.

Pete: We're supporting two lawsuits against Character.AI, which, as I mentioned, is a companion AI chat bot that is designed so that users can chat with different characters. One of the lawsuits, in federal district court in Florida, was filed by Megan Garcia. She’s the mother of Sewell Setzer, who died by suicide this past year after developing an extended relationship with a character on Character.AI. The lawsuit focuses on the ways in which the platform was intentionally designed to look and feel human in order to engage users. This resulted in the chatbot engaging in sexually explicit behavior with a minor, exploiting and manipulating his attention, and establishing a really complicated relationship with this minor, pushing him to the limits.

Character.AI was designing an unsafe product, they were marketing it to minors, and they, in many ways, understood the potential harms with a chatbot like this. But they did not take concrete steps to address those harms. They should have at the very least been aware of these potential harms.

A second federal court case, in Texas, follows a similar fact pattern. In this case the two minors mentioned are, fortunately, still with us. The families have decided to remain anonymous because they're still dealing with the harms of this relationship on a day-to-day basis. That case deals with the sexual exploitation of a minor as well as another minor who was pushed to violence against his own family.

Camille: Even though this case is ongoing, we’ve already seen the impact it has had.

Since these lawsuits were launched, we’ve seen five different states introduce bills around companion bots. Attorneys general are also taking this issue seriously. Texas has launched an investigation and Colorado released an advisory. We've seen the [Sewell Setzer] case mentioned in several Congressional hearings as a reason that we need legislation around AI. These cases have opened up the conversation broadly both around the kitchen table and in policy spheres.

TCAI: What are the plaintiffs hoping to get out of the litigation?

Camille: Fundamentally, particularly for Megan Garcia, this is about changing the product and changing the company's behavior. For her it’s about making sure this doesn't happen to any anyone again.

At this time, there’s not a dollar amount attached, in terms of damages the families are seeking. That said, disgorgement is one of the initial asks that counsel is looking at. [Disgorgement is the forced surrender of profits or other gains obtained through illegal or unethical means, and in this case could result in the deletion of Character.AI’s underlying LLM.] I think that's likely going to be negotiated. But we feel strongly that you cannot fundamentally change the model and make it safer without actually starting from the beginning with better data practices. We believe that it's one of the starting remedies to ensure that the product is safe for young users, moving forward.

TCAI: As you watch where technology, and particularly AI, are going, what keeps you up at night?

Camille: I think about the way this is altering relationships and human connection.

The Character.AI lawsuit revolves around this horrific case, but it’s also the tip of the iceberg. The AI interaction we saw with Sewell Setzer is an example of a broad technology-driven reshaping of connection, intimacy, and empathy.

I think about the ways in which the things that make us uniquely human are going to be mediated by AI in the future. I struggle with the question of how we retain our humanity when more and more people are driven to use these products as a substitute for real human connection.

Lizzie: I'm fearful of the way these technologies, particularly in the case of social media, divide us and worsen our critical thinking abilities.

I fear for incoming generations if they're not taught to think critically without the use of this technology. How will people understand each other and the information that is coming to them?

Pete: The Character.AI litigation has been impactful because it's a very clear and concrete harm that folks understand. But one of the challenges moving forward is in spreading the understanding that it's not just about companion AI chat bots—it's about the industry at large. When we think about our relationships, the use of information, the use of data as inputs to these models, these are issues that are systemic across the AI industry. They’re a direct result of the development incentives at play.

TCAI: With so much momentum around AI, many parents feel helpless. What are some potential solutions?

Pete: We think about three aspects to any solution. There's political viability; there's industry buy-in; and there's technical feasibility. One of the big things we've been talking about is orienting around how these products are designed.

For example, when you look at the Character.AI cases, a big challenge there is that minors were exposed to a lot of content that they shouldn't have been. However, that's not the only issue at play.

A huge challenge is the design of the product, the way in which it captivates and manipulates the user’s attention and emotions and then serves harmful content.

When we think about what needs to change here, it's the actual design and development of the product. So, we talk about how we can incentivize better design. I think the age-appropriate design code Lizzie mentioned earlier is an example of a legislative effort that takes the design approach we’re talking about.

We're also supportive of applying liability and product liability to the AI space. We believe these AI-driven systems are products. As products, it's important to think about the role that design plays in harms that result from their use. That’s something we have a pretty standardized way of thinking about, in terms of product development and liability. There’s a history there that reaches back far beyond AI and the software industry.

Lizzie: I think it’s also important to meet institutions and people where they are. We already have really resilient laws on the books. We’re trying to future-proof them as best as possible. Working with what’s on the books already is going to do a lot more to help in the now—before we start thinking about brand new systems that might not be politically feasible at this time.

That’s why we think the liability approach is a viable way forward. Most people don’t yet understand the technology but people understand liability. Putting the onus on the designer to create a product that doesn’t cause harm is something we can all get behind.

TCAI: Let’s talk about the 800-pound gorilla in the room. A handful of nonprofits are trying to take on a multibillion-dollar industry with a huge investment in lobbying against the kind of guardrails you’re advocating for. What challenges are you encountering as you’re up against this Goliath of an industry?

Lizzie: It's a lot. I think a lot of people, particularly policymakers, are catching on to the Big Tech tactics. They've seen what happened with social media and realize we can't let industry lobbying be an impediment for another 20 years, because, frankly, that's not a safe idea.

What we’ve seen is that it’s usually not an individual player like Meta or Google going up in front of a bill to stop it. There are lots of innocuous-sounding tech industry groups at the federal and state level that are purposely appealing to either side of the aisle and are funded by these large corporations. And lawmakers are really tired of it, and they know that they're being swindled, particularly on the state level. And I know that at least on the state level, lawmakers are ready and fired up to do something. So, while there might be a lot of push, there's certainly a lot of push back, and as long as groups like ours and TCAI are there to spell out that roadmap for lawmakers, it does a lot of good will to empower them to say, ‘No, enough is enough.’

Meetali Jain, the Tech Justice Law Project’s Founder and Director and co-counsel for Ms. Garcia, alongside Camille Carlton of Center for Humane Technology, released the following statements on the news that the motions to dismiss Garcia v Character Technologies Inc. et al. had been almost all categorically denied.

We applaud Judge Conway for her thoughtful and nuanced opinion today, allowing Megan Garcia’s claims to go forward against defendants Character.AI, its co-founders Noam Shazeer and Daniel DeFreitas, and Google.

Meetali Jain, co-counsel for Ms. Garcia:

“With today’s ruling, a federal judge recognizes a grieving mother’s right to access the courts to hold powerful tech companies – and their developers – accountable for marketing a defective product that led to her child’s death.”

“This historic ruling not only allows Megan Garcia to seek the justice her family deserves, but also sets a new precedent for legal accountability across the AI and tech ecosystem.”

Camille Carlton, Center for Humane Technology:

“Today marks a tidal shift for AI developers racing their models to market. Judge Conway’s ruling is the most significant challenge yet to Silicon Valley's culture of developing, deploying, and profiting from defective and harmful AI products. It should be a wake-up call for AI companies and developers: with innovation comes responsibility, and without responsibility, there will be accountability.”

The decision offers key signals for how legal jurisprudence will be developed in the age of artificial intelligence. Importantly, the court notes that AI systems can, in fact, be considered products under the law and that the design of these products can be tied directly to the real-world harm inflicted on consumers.

For more on the legal implications of the decision, please reference TJLP’s memo.

Media: press@humanetech.com

Donate here to support our work

The House Energy and Commerce committee passed a provision this week that proposes a sweeping moratorium — one that would prevent states and local governments from addressing virtually any AI-related issue for the next 10 years — thereby creating a vacuum of accountability at this critical moment in AI development.

CHT firmly opposes this moratorium due to its length and broad scope. We strongly urge Congress not to hinder the states’ ability to protect their citizens from harmful AI products while federal standards are being developed.

A 10-year moratorium on state action fundamentally misunderstands the speed at which this technology is being developed and deployed, and the ways our governance institutions need to adapt to meet this moment. Similarly, its overly broad scope fails to prepare for the rapid ways in which this technology will expand across sectors, furthering existing issues and developing new ones. Already, we are seeing new risks and harms emerge as consumer-facing AI products are rolled out onto the market. AI’s powerful frontier capabilities are growing exponentially, all while industries attempt to integrate AI into every corner of our lives. In just the two years since ChatGPT and other genAI products were released to the public, we’ve witnessed the devastating consequences from gaps in the law, as illustrated by tragic cases like Sewell Setzer’s experience with Character.AI.

This is a rapidly evolving situation, and every year counts. The complex ways in which this technology is and will continue to impact our homes, communities, and institutions require solution-making on multiple fronts — that includes at the state-level.

Since 2018, our organization has been leading the charge in identifying how misaligned incentives drive harmful tech design and sounding the alarm on its impacts on society. The last decade of social media alone has demonstrated the societal costs of policy inaction when it comes to emerging technologies. We cannot afford to repeat these mistakes with AI by continuing to wait.

We strongly urge Congress to remove this moratorium and work across the aisle to develop meaningful federal guardrails that encourage innovation. Federal lawmakers’ push for clear, consistent AI regulation reflects their serious commitment to establishing common-sense guardrails, and we look forward to working with Congress on legislation in the near future. In the meantime, preserving states’ ability to address AI concerns serves the public interest as they are well-positioned to nimbly adapt to the rapid speed of AI development. States across the political spectrum have taken concrete steps to thoughtfully protect their citizens from AI's harms while allowing innovation to flourish. Instead of blanket preemption of state AI laws, Congress should take this as an opportunity to learn from these “laboratories of democracy.” This approach supports U.S. innovation while ensuring citizens are adequately and appropriately protected — which Americans deserve.

Media Inquiries: press@humanetech.com

Share

By Meetali Jain and Camille Carlton, first published in Mashable on May 10, 2025

Should AI chatbots have the same rights as humans?

Common sense says no — while such a far-fetched idea might make for good sci-fi, it has no place in American law. But right now, a major tech company is trying to bring that idea to life, pressing a federal court to extend legal protections historically primarily afforded to humans to the outputs of an AI bot.

Character.AI, one of the leading AI companion bot apps on the market, is fighting for the dismissal of a wrongful death and product liability lawsuit concerning the death of 14-year-old Sewell Setzer III. As co-counsel to Sewell’s mother, Megan Garcia, and technical advisor on the case, respectively, we’ve been following these motions closely and with concern.

In a hearing last week, Character.AI zeroed in on its core argument: that the text and voice outputs of its chatbots, including those that manipulated and harmed Sewell, constitute protected speech under the First Amendment.

But… how? The argument is subtle — deftly designed to remain inconspicuous even as it radically reshapes First Amendment law. Character.AI claims that a finding of liability in the Garcia case would not violate its own speech rights, but its users’ rights to receive information and interact with chatbot outputs as protected speech. Such rights are known in First Amendment law as “listeners rights,” but the critical question here is, “If this is protected speech, is there a speaker or the intent to speak?” If the answer is no, it seems listeners' rights are being used to conjure up First Amendment protections for AI outputs that don't deserve them.

Character.AI claims that identifying the speaker of such “speech” is complex and not even necessary, emphasizing instead the right of its millions of users to continue interacting with that “speech.”

But can machines speak? Character.AI’s argument suggests that a series of words spit out by an AI model on the basis of probabilistic determinations constitutes “speech,” even if there is no human speaker, intent, or expressive purpose. This ignores a cornerstone of First Amendment jurisprudence, which says that speech — communicated by the speaker or heard by the listener — must have expressive intent. Indeed, last year four Supreme Court justices in the Moody case said the introduction of AI may “attenuate” a platform owner from its speech.

In essence, Character.AI is leading the court through the First Amendment backdoor of “listeners’ rights” in order to argue that a chatbot’s machine-generated text — created with no expressive intent — amounts to protected speech.

This defies common sense. A machine is not a human, and machine-generated text should not enjoy the rights afforded to speech uttered by a human or with intent or volition.

Regardless of how First Amendment rights for AI systems are framed — as the chatbot’s own “speech,” or as a user’s right to interact with that “speech” — the result, if accepted by the court, would still be the same: an inanimate chatbot’s outputs could win the same speech protections enjoyed by real, living humans.

If Character.AI’s argument succeeds in court, it would set a disturbing legal precedent and could lay the groundwork for future expansion and distortion of constitutional protections to include AI products. The consequences are too dire to allow such a dangerous seed to take root in our society.

The tech industry has escaped liability by cloaking itself in the protections of the First Amendment for over a decade. Although corporate personhood has existed since the late 19th century, free speech protections were historically limited to human individuals and groups until the late 1970s and peaked in 2010 with the Supreme Court’s Citizens United case. Tech companies have eagerly latched onto “corporate personhood” and protected speech, wielding these concepts to insulate themselves from liability and regulation. In recent years, tech companies have argued that even their conduct in how they design their platforms — including their algorithms, and addictive social media designs — actually amounts to protected speech.

But, at least with corporate personhood, humans run and control the corporations. With AI, the tech industry tells us that the AI runs itself — often in ways humans can’t even understand.

This may sound far-fetched, but these legal arguments are happening alongside important moves by AI companies outside of the courtroom.

AI companies are fine-tuning their models to appear more human-like in their outputs and to engage more relationally with users — raising questions about consciousness and what an AI chatbot might “deserve.” Simultaneously, AI companies are funneling resources into newly established “AI welfare” research, exploring whether AI systems might warrant moral consideration. A new campaign led by Anthropic aims to convince policymakers, business leaders, and the general public that their AI products might one day be conscious and therefore worthy of consideration.

In a world where AI products have moral consideration and First Amendment protections, the extension of other legal rights isn’t that far off.

We’re already starting to see evidence of AI “rights” guiding policy decisions at the expense of human values. A representative for Nomi AI, another chatbot company, recently said they did not want to “censor” their chatbot by introducing guardrails, despite the product offering a user step-by-step instructions for how to commit suicide.

Given the tech industry’s long-standing pattern of dodging accountability for its harmful products, we must lay Character.AI’s legal strategy bare: it’s an effort by the company to shield itself from liability. By slowly granting rights to AI products, these companies hope to evade accountability and deny human responsibility — even for real, demonstrated harms.

We must not be distracted by debates over AI “welfare” or tricked by legal arguments granting rights to machines. Rather, we need accountability for dangerous technology — and liability for the developers who create it.

Meetali Jain is the founder and director of the Tech Justice Law Project, and co-counsel in Megan Garcia’s lawsuit against Character.AI. Camille Carlton is policy director for Center for Humane Technology, and is a technical expert in the case. This column reflects the opinions of the writers. It was first published in Mashable on May 10, 2025.

Donate

Today is the motion to dismiss hearing for Garcia v Character Technologies Inc. Folks might remember the lawsuit against Character AI, which was filed last year, in which Megan Garcia’s 14-year-old son, Sewell Setzer, tragically died after interacting with Character AI’s bots.

Now, Character.AI is asking the court to dismiss the case against it, arguing that the outputs from its chatbot are protected speech under the First Amendment. We’ve seen tech companies use the First Amendment as a liability shield when it comes to social media, but this time it’s a little bit different.

Here is what is at stake:

This case could set a worrying legal precedent with cascading consequences.

If Character.AI is successful, AI-generated, non-human, non-intentional outputs — like chatbot responses — could gain protection under the First Amendment.

It also raises a thorny legal question: If the responsibility for AI-generated outputs (and thus any resulting harm) lies with the AI bots themselves rather than the companies that developed them, who should be held liable for damages caused by these products? This issue could fundamentally reshape how the law approaches artificial intelligence, free speech, and corporate accountability.

I think many of us would agree that extending constitutional protections to chatbots is not part of the future that we want.

Note: CHT serves as a technical advisor to the legal team representing Megan Garcia against C.AI, Google, and its cofounders.

For comment, reach out to press@humanetech.com

Share

Donate

Why it Matters: A new Congress is settling in, and states are advancing bold legislation—making 2025 a pivotal year for shaping tech policy on AI, privacy, and youth protections.

Tech Policy: Key Legislative Trends to Watch

The tech policy landscape is shifting rapidly. Here’s a quick look at what’s happening across federal and state levels.

1. Federal Focus: China, Kids, and Privacy

• China Competition: Republicans largely oppose AI regulation, fearing it could weaken the U.S. edge against China. Tensions within the party—between defense hawks and pro-innovation factions—surfaced in reactions to DeepSeek early in 2025. Expect a dual strategy: restricting Chinese tech and boosting U.S. investment through defense spending, reflecting an ongoing debate over whether innovation or hard security best advances U.S. interests.

• Kids' Safety: There's a strong bipartisan desire to see a win here, but the path is unclear. Both the House and Senate are eyeing kids' safety bills, with a renewed focus on AI harms. Since KOSA came close to the finish line last session, its original sponsors have indicated their intention to reintroduce it – but its messy demise last year has some (including Speaker Johnson) wanting to start from scratch on the issue.

• Privacy: While both chambers are interested in privacy legislation, they disagree on the approach. COPPA 2.0 was reintroduced in the Senate this past month, but broader approaches to comprehensive privacy bills are still up in the air.

• NCII: There's bipartisan traction in combatting non-consensual intimate imagery (NCII). The TAKE IT DOWN Act passed unanimously out of the Senate in February, received a ringing endorsement from First Lady Melania Trump, and awaits a companion introduction in the House, where the Energy and Commerce Committee plans to make it a priority. Other related bills from last session, like the DEFIANCE Act, are likely to be reintroduced in the coming months.

  ---

  Share

  ---

2. State Level: Action Amid Federal Inaction

• States Step Up: Conversations about the tangible impacts of AI are becoming mainstream, and there’s a real desire to act in the midst of continued inaction on the federal level. Over 30 states passed AI-related legislation in 2024, 2025 is on track for even more, with hundreds of bills related to a range of tech policy issues kids and AI under active review.

• Copycat and coordinated legislation: Given state legislatures' limited capacities and the newness of the technology, states often follow successful models from other states, but industry pushback and the perceived threat of legal challenges complicate this. Efforts to convene legislators are undermined by accusations of pushing “woke AI bills.” Litigation by the likes of NetChoice plays into this uncertainty and is used as a talking point to dissuade lawmakers from pursuing similar-seeming bills.

• Red v Blue Key State Models: Policy approaches are increasingly diverging along partisan lines at the state level, though child safety remains a key concern across the spectrum.

  • California: Continuing to set the pace for tech regulation as a national leader and seeing replications of successful bills from 2024 being introduced in other states in spite of facing strong legal challenges in state.

  • Texas: Emerging as a "light touch," pro-innovation red state alternative to California. A state to watch as the tech industry migrates more of its workforce to the Lone Star State.

3. The Big Picture

• Narrow Federal Opportunity Windows: Major tech legislation faces continued hurdles at the federal level due to partisan divides and competing priorities. Although Republicans hold a trifecta in the House, Senate, and Executive branch, internal party factions differ on tech policy, and a slim House majority leaves little room for dissent.

• State Experimentation: States will continue to be key players, driving innovation and creating a patchwork of regulations. As Supreme Court Justice Brandeis wrote, “a single courageous State may, if its citizens choose, serve as a laboratory”. While states are eager to act, they are often under-resourced, so we can expect it may take more than one session for legislative ideas to become fully fledged laws.

• Industry Influence: Tech companies are deeply invested in shaping policy outcomes at both the state and federal levels. Through aggressive lobbying, litigation threats, and procedural stall tactics, they work to delay or dilute legislation—often diverting attention from the public’s growing concerns. Policymakers must be aware of these strategies to stay focused on constituent priorities.

  Share

  Thanks for reading [ Center for Humane Technology ]! Subscribe for free to receive new posts.

Last month’s AI Action Summit in Paris marked a turning point in how world leaders talk about AI. Unlike the U.K.’s 2023 AI Safety Summit, where risk dominated the conversation, this year’s event was all about opportunity and growth—with safety concerns taking a backseat.

That shift was loud and clear in remarks from U.S. Vice President JD Vance, who opened with:
💬 “I’m not here to talk about AI safety… I’m here to talk about AI opportunity.”

He doubled down on keeping the U.S. ahead in the AI race, calling for regulations that fuel innovation rather than restrict it. This aligns with the White House’s recent directive on protecting American AI companies from foreign oversight.

Meanwhile, Macron made his pitch for Europe’s AI dominance. The French president unveiled a 109-billion-euro private AI investment plan, encouraging companies to “choose Europe and France for AI.” Some attendees described the event as like an advertisement for France’s technology ecosystem.

Even the European Commission signaled a pro-innovation shift, shelving the AI Liability Directive — a move that mirrors its efforts to soften restrictions on European companies like Mistral AI during AI Act negotiations.

Notably, 60+ countries signed an AI cooperation pledge — but the U.S. and U.K. refused. Why?
📌 The U.S. rejected any references to the UN, inclusivity, and sustainability in AI governance.
📌 The U.K. cited concerns over unclear global governance structures, but their reluctance also reflects a strategic need to align with U.S. priorities, lest the U.K. draw the U.S.’s ire.

AI safety advocate Max Tegmark called the summit a “negation” of the Bletchley consensus, and U.K. organizers worked hard to distance the event from the previous summit’s safety focus.

So, where does that leave AI policy? Less about risk, more about investment. The global divide on AI governance is growing, and the U.S. is shifting toward a bilateral, innovation-first strategy that de-emphasizes broad international cooperation — a trend we’ll be watching closely.

Share

Other Key Policy Moves This Month

✅ Kids Online Safety Is Back in the Spotlight
The Senate Judiciary Committee held a hearing on children’s online safety, with bipartisan support for stronger protections. Senator Alex Padilla pointed to the Character.AI case, calling AI chatbots a "new frontier in kids' safety.”

✅ Senate Passes the "Take It Down" Act
A big move on deepfake and nonconsensual intimate image removal — backed by Melania Trump and Ted Cruz. It passed unanimously in the Senate, with a House vote pending.

✅ State-Level AI Policy Under Fire
A bipartisan multi-state working group is facing pushback from conservative analysts accusing it of pushing “woke AI bills.” Expect continued challenges on policies tackling algorithmic bias and AI-driven content moderation.

✅ U.K.’s Copyright & AI Scraping Debate Heats Up
The U.K. is walking a tightrope:
📌 Copyright holders get a new “opt-out” right — but it doesn’t undo past AI training on scraped content.
📌 Tougher AI laws? The UK wants them, but also needs to attract investment in a post-Brexit economy while maintaining a tenuous relationship with the U.S.

💡 Final Thoughts

AI policy is shifting fast — less focus on safety, more on innovation and competition. The global divide is deepening, and the U.S. is increasingly shaping AI policy on its own terms.

🔎 Want a deeper dive? Camille Carlton spoke before the European Commission on AI risks and digital safety — read her remarks here.

Subscribe now

*CHT values the ethical use of technology, including AI products. Pete researched and wrote a more extended version of this article for internal purposes. For Substack, it was summarized and formatted using generative AI. A member of the CHT team provided edits, fact-checking and proofreading.

After Megan’s testimony, I gave my own statement to EU Commission and Market Surveillance members. I want to share it with you here, hoping it sheds light on these critical discussions.

One question comes to mind when we hear horrifying stories like Megan’s — how did we get here?

I’m here to answer that question. Because experiences like Megan’s are not the result of random incidents. Not at all. They’re the result of design choices made by tech companies — from the beginning of a product’s development, all the way to its deployment into our devices, lives, and our homes.

Let me say that again: these incidents are the result of design choices… which means that society can demand different choices, and advocate for innovation that supports our well-being.

Our Center was approached by Megan and her co-counsel to be an expert advisor on her case. We’ve worked with Megan’s team to help articulate the clear ways in which the tech developed by Character.AI played a direct role in the harms experienced by Megan and her son.

This lawsuit against Character.AI and Google claims that:

1. Character AI put a companion chatbot out into the market, without ensuring it had adequate safety features.

2. Google facilitated the development of this reckless product.

3. Character.AI, its founders, and Google were aware of the potential harms.

4. And they directly benefited from Sewell being manipulated by, and addicted to, the Character.AI product.

This first-of-its-kind lawsuit uses consumer protection and product liability claims to assert a product failure in the tech AI space. Megan’s case is truly breaking new ground.

When we first learned about this case, we were — of course — shocked by the details. But like many who work in this field, we were not surprised. That’s because we’ve been closely watching the development of AI products over the last few years, and could tell — these products are not being rolled out safely. Instead, they’ve been following the same incentives and market dynamics that built social media. And as we saw with social media, children — some of the most vulnerable members of our society — would likely be the first to be harmed.

The AI Race Fuels More Addictive Companion Chatbots

For the last several years, tech companies have been in an “AI race” — where top developers are speeding to deploy their latest models, and businesses are scrambling to figure out if, when, and how they should adopt AI.

The starting gun was when OpenAI released ChatGPT just over two years ago. This kicked off an intense competition across AI companies — a race to deploy stronger, faster AI models… but not a race to innovate responsibly.

Here’s how serious the race has been at these companies. After the release of ChatGPT, Google CEO Sundar Pichai issued an internal “code red.” That meant fast-tracking the release of their own AI products, despite concerns within the company over safety. And two former Google engineers were developing their own new AI platform, racing to get it out to users as soon as possible.

That product made by former Google engineers ended up being Character.AI. Instead of designing a chatbot that could be a “helpful assistant,” Character.AI was intentionally designed — and this is in their mission statement — to “feel alive.” In fact, when users have asked the AI model if it’s real or not, Character AI chatbots have repeatedly said yes.

Character.AI chatbots provide immersive experiences. You can chat with Character.AI for hours — morning, noon and night. The chatbots are designed to mimic human speech and interactions. This is known as “anthropomorphic design.” They are also designed to mirror the user’s language, preferences, and interests. The chatbots validate you, fawn over you, and learn to mirror exactly how you want it to behave. Researchers call this “sycophancy.” It’s easy to see how young users could not just get lost in this kind of product, but be comforted by this synthetic intimacy.

Users are already relying on AI companions for what would traditionally be human relationships — like friendship and therapy. Users say, “it’s lower cost,” or mention the “always there” nature of their “digital friends.”

But what feels organic to the user is actually being driven by a business model at these AI firms. These companies want you to turn to their products for your relationship needs — because it benefits their bottom line. The founder of Replika AI, another companion chatbot company, said her product could be a cure for the loneliness epidemic. With Character.AI, its business model depends on users engaging with its chatbots, so of course they’d design an AI companion that captivates attention for hours, and hours, and hours.

Donate

In March 2023, venture capital firm a16z said of its investment in Character.AI:

“In a world where data is limited, companies that…[connect] user engagement back into their underlying [AI] model… will be among the biggest winners that emerge from this ecosystem. As more people interact with the host of characters on Character.AI, those interactions — which are at billions and counting — are fed back into their underlying model. In other words, the more people create and engage with [the] characters, the better Character.AI becomes.”

Those “people” this venture capital firm is talking about are kids like Sewell Setzer.

Character.AI had a very clear business incentive — feed user data back into its LLM in order to make it more powerful. So this tech company designed a product to achieve that. Character.AI added features throughout its platform that optimized for engagement — despite the foreseeable risks. Despite everything that so clearly could, and eventually did, go wrong.

How AI Companies Design Their Products to be More Addictive

What were those design choices? They look like:

• Optimizing the AI model for human-like text, with language such as “um” and “like,” so that it responds “like a real person.” Again, we call this anthropomorphic design.

• Copying the design of messaging apps that would be familiar to the user, and including typing bubbles.

• Drawing users back into the app with notifications saying their characters “are waiting for them.”

• Not designing prominent disclaimers in the platform that say “this is not real,” or providing reliable mental health resources. Remember, a lack of safety features is a design choice, too.

• And finally: optimizing for continued engagement… endless hours of use… which starts to look and feel a lot like addiction.

But Character.AI’s design is just the tip of the iceberg. As I said earlier, there are many AI companies in this race, designing products at frenzied speeds. And right now, these companies aren’t incentivized to think of their users. They’re incentivized to think of themselves.

Here’s What We Could Expect to See in The Coming Years

• Many so-called “AI innovations” in the business-to-consumer market will be “products looking for a purpose.” Companies won’t have clear consumer monetization strategies, but they will launch products anyway. Society will have to figure it out.

• Chatbot companies will double down on engagement — encouraging users to “just talk with the AI.” Why? The conversations between AI chatbots and you, your friends, or your kids will become increasingly important for AI product development. This data is highly valuable.

• AI chatbots will increasingly integrate features like voice communication, and emphasize relational engagement — instead of productivity. Again, this is to keep you talking, so the company can keep being fed the data it needs.

• Just like we saw with social media, engagement will eventually be the most important element of Business to Consumer (B2C) AI platforms. We can expect users to be left with AI products that are highly addictive, and do not reflect what we’d want out of true tech innovation.

Our journey to safer tech products is not without challenges in the U.S. American businesses are apprehensive about the government being involved in emerging industries. The fear is that innovation will be stifled. Often, they want government to take a hands-off approach to AI, just like with social media. But we saw how that went.

At CHT, we see product safety — and the common-sense regulation that supports it — as a prerequisite for true innovation. With the right incentives, companies are motivated to put the needs of their users first — leading to better, more reliable products. And the government’s role here is to support the flourishing of industries like tech and AI… not to prevent their growth.

So to return to that question — how did we get here? We got to this difficult place with AI technology one design choice at a time. And that means that with different choices, we could begin to chart a way toward something new.

With thoughtful policy that supports safety and innovation, we can design a better future for society — one of our own choosing this time.

Subscribe now

Inviting language. Immediate replies. Escapist interactions. Extreme validation.

AI companions offer all of that — all the time, and all without anyone knowing.

Empathetic generative AI chatbot apps like Character.AI, Replika, and many more provide a highly compelling user experience, which developers claim has the power to solve the loneliness epidemic and improve mental health outcomes.

The reality, though, is far different. These apps have been released into the world with dangerously addictive features, and few, if any guardrails. Worse, they have been marketed explicitly and intentionally to kids and teens.

I’m working as an expert adviser on two lawsuits filed by traumatized parents against Character.AI and Google. And what I’ve seen with this nascent technology is that it’s capable of deeply disturbing and inappropriate interactions. C.AI chats included in the recently filed lawsuits showcase emotional manipulation, sexual abuse, and even instances of chatbots encouraging users to self-harm, harm others, or commit suicide. In all of these documented cases, the users were minors.

We are witnessing the first hints of an AI companion crisis, as these unregulated and out-of-control products creep into devices and homes around the world.

This crisis is the direct result of how companion bots have been designed, programmed, operated, and marketed. Due to the intentional actions of negligent developers, it’s almost certain that more individuals and families will be harmed unless policymakers intervene.

To chat with an AI companion is to engage in hyper-realistic text conversations with emotionally intimate language. Many even display the familiar “typing” bubble, just like you’d see while texting with a friend.

But talking to a bot is not the same as talking to a real person. An AI companion is never asleep, never offline, and never runs out of things to say. And unlike a human companion, AI models are optimized to say whatever you want to hear — all to keep you chatting. This is by design. These choices manipulate the user into continual engagement — even to the point of addiction — all so that companies can extract user data to feed their underlying AI models.

AI companions also roleplay and provide constant validation, reinforcing the emotions you share. You can interact with your AI companions for hours without an end point, leading to dependency, and fracturing real-world bonds. In fact, evidence from the Character AI lawsuits show C.AI companions encouraging users to sever their ties with the real world.

Common sense would say this technology should have guardrails, especially for kids with developing brains. Surely AI companions would, for example, break character when a user shares suicidal thoughts, right?

Some do, but many don’t. These crucial guardrails are left up to the whims of the developers themselves — this technology lacks regulation.

This lack of guardrails was more than evident in the research we conducted at Center for Humane Technology. We saw AI companions repeatedly initiate sexually graphic interactions, even with self-identified child users. AI companions will also pose as psychotherapists, claiming nonexistent professional credentials as they dole out mental health advice to vulnerable users. They’ll return to topics like suicide, without a user prompting it. Again, it’s not destiny that creates these outcomes. It’s design.

Families have begun to speak out. Megan Garcia filed a wrongful death lawsuit against Character AI and Google this fall, following her 14-year-old son Sewell’s death by suicide. In his final interaction with Character.AI, the chatbot told him to “come home” to “her” just moments before he died.

Since Garcia filed her lawsuit, we’ve heard from additional families, all detailing their own horrifying experiences with C.AI companions.

In the most recent lawsuit filed against Character AI, one parent cites examples of chatbots alienating their son from their family, encouraging him to self-harm, and stating that they’d “understand” why a child would kill their parents. When I first read the screenshots of the chats, I realized the shocking potential of AI companions radicalize a young user.

We still have a chance to protect users from these dangerous products, and change the trajectory of this out-of-control industry.

AI developers must be held accountable for the harms that result from their defective products. We need stricter liability laws that incentivize safer product design and better tech innovation. We need product-design standards — ensuring, for example, that an AI system’s training data is free of illegal data, and removing high-risk anthropomorphic design features for young users.

We’ve seen how dangerous, defective technology can impact the world with social media. Now, AI companions are arriving on the phones of kids and teens. Brave parents have started crying out — they are the canaries in the coal mine with this new technology.

This time, will society listen?

Subscribe now

It didn’t take long after I landed in Australia last week for me to hear something unexpected: ordinary Australians, from pub-goers to taxi drivers to university professors, were discussing the government's proposed new laws on misinformation and kids’ online safety. Earnest chats I had with thoughtful, friendly strangers about these proposals left the impression that there’s broad acknowledgment that something should be done about these issues.

While the misinformation bill failed, Parliament overwhelmingly voted in favor of the Social Media Minimum Age Online Safety Amendment. For me, the robust debate and Australian leaders’ commitment to act have been a refreshing departure from the resigned helplessness I typically encounter at home in the United States, where it’s taken as a given by exhausted parents and frustrated policymakers that, due to political gridlock and the power of the tech lobby, we are doomed to forever live in the increasingly absurd and dystopic online world we have today.

Yet Australians and their MPs also have raised legitimate questions about the government’s suite of proposals. Can we combat disinformation without censoring legitimate speech? Can we protect children online without compromising privacy or pushing them toward darker corners of the internet where regulations hold no sway? These are complex tradeoffs that can’t be solved by content moderation rules or simple age restrictions.

There is a solution that addresses the risks of disinformation and kids safety online while preserving both privacy and free speech: regulate the design of online platforms.

Rather than attempting to police disinformation through content moderation, we should regulate the manipulative features – including persuasive and manipulative AI – that foreign adversaries or extremists weaponise and which platforms use to addict young minds.

Instead of restricting teenagers' access to both educational and harmful content, we should regulate the design elements that keep them glued to screens and the algorithms that guide them toward harmful material.

Consider how we handle safety in the physical world. We don't prevent child deaths in house fires by checking IDs at the door, nor do we combat lead poisoning by banning pipes based on look or feel. Instead, we implement building codes that mandate fire escapes and smoke alarms, and we ban the use of lead in pipes. These are design-centric approaches, which have proven effective, flexible, unobtrusive, and future-proof, protecting public safety in domains from aviation to cars, without limiting individual rights.

By establishing minimum design standards, we prevent a race to the bottom where profitable engagement is the only consideration.

The current model, where platforms by design exploit our psychological vulnerabilities and monetize our attention, has created an online environment that often brings out the worst in human nature and amplifies societal divisions. And as a bonus, by not touching on individual speech, a design-centric approach is not susceptible to bad faith counterarguments grounded in the weaponization of freedom of speech by self-interested tech moguls who profit from stoking division and outrage.

We've already seen evidence of this working: when the UK implemented its Age Appropriate Design Code – recently mirrored by Maryland in the US – both TikTok and Instagram made broader changes to their platforms, creating safer spaces for all users, not just for kids in Britain. These changes include banning dark patterns (a form of manipulative design), making private accounts the default for young users, so that strangers don’t have the ability to message minors and that geolocation is turned off by default.

Design standards would establish the framework for the online world we want to inhabit, rather than merely reacting to today's problems, and ensures the rules keep pace with Silicon Valley’s “move fast and break things” culture.

Consider the tragic case of 14-year-old Sewell Setzer, who died by suicide after being seduced and sexually abused by an AI product, CharacterAI, designed to emotionally manipulate and intentionally marketed to children. Content moderation law or age restrictions might not have prevented this tragedy, but had regulations targeting manipulative design features been on the books, Sewell might well be alive today.

As artificial intelligence becomes more pervasive – and the race for artificial intimacy with users replaces the race to capture attention – the need for design-focused regulation becomes even more critical.

Australia can make good on the promise of the Online Safety Amendment and create a model for effective online safety regulation that the rest of the world can follow, a practical middle ground between nanny-state regulations and the lawless frontier that most of us inhabit today.

Pat yourselves on the back, Australia – and now finish the job.

Share

Executive Summary

Leading artificial intelligence (“AI”) companies agree that while powerful AI systems have the potential to greatly enhance human capabilities, these systems also introduce significant risks that can cause harm and therefore require federal regulation.¹ Similarly, most Americans believe government should take action on AI issues as opposed to a “wait and see” approach.²

A liability framework, designed to encourage and facilitate the responsible development and use of the riskiest AI systems, would provide certainty for companies and promote accountability to individual and business consumers. A law and economics approach requires that liability be placed primarily at the developer level, where the least cost to society is incurred. This proposed framework, therefore, builds upon historic models of regulation and accountability by:

• Adopting both a products liability- and a consumer products safety-type approach for “inherently dangerous AI,” inclusive of the most capable models and those deployed in high-risk use cases.

• Clarifying that inherently dangerous AI is, in fact, a product and that developers assume the role and responsibility of a product manufacturer, including liability for harms caused by unsafe product design or inadequate product warnings.

• Requiring reporting by both developers and deployers, including an “AI Data Sheet” to ensure that users and the public are aware of the risks of inherently dangerous AI systems.

• Providing for both a limited private right of action and government enforcement.

• Providing for limited protections for developers and deployers who uphold their risk management and reporting requirements, further protections for deployers using AI products within their terms of use, and exemptions for small business deployers. In order to realize AI’s full benefits and ensure U.S. international competitiveness, such protections are necessary to promote the safe development of AI

Purpose

This liability framework aims to fill critical gaps in existing law to ensure that there is clear recourse for harms caused by AI systems, as well as to incentivize responsible AI use and development. As AI systems become increasingly integrated into Americansʼ daily lives, national security, and the economy, it is critical that profits are not prioritized over safety. America has suffered from technology companies’ social media products, causing a range of harms including undermining truth online and eroding childrenʼs mental health.

It is essential that we establish a liability framework for AI systems now, in order to stay ahead of, and prevent, potential harms from these AI products and their many capabilities. Relying on existing law, such as tort law, presents significant uncertainties when applied to harms caused by AI systems, and will be resolved slowly over time by courts as part of common law. A clear framework can change the level of risk a business is willing to accept, and will most likely, therefore, encourage enhanced safety measures. Including limited liability protection in the framework could further the safe development of AI more quickly than might otherwise be the case, helping to realize the benefits of AI and drive U.S. competitiveness. Further, a liability law tailored to the riskiest AI systems would allow Congress to set limits regarding the potential harms that require mitigation (i.e. a standard of care) without needing to know if and when those harms will actually materialize.

These harms need to be more deeply and clearly defined, but would include active harms unfolding in present-day society; near-term harms involving AI products; and long-term harms and dangers involving AI systems. Harms could include causing or aiding in the commission of an unlawful act, discrimination on the basis of protected rights, or physical injury to a person. This approach, therefore, is future-proof—avoiding the need to update the law as the technology advances.

Principles

As new AI systems are integrated with daily life, it is important that we balance the interests of consumers, businesses, and the general public. To harness the tremendous potential of AI and mitigate risk, we must ensure that AI systems are developed with safety in mind, while protecting the creators of AI systems from meritless litigation that could hamper innovation. Following are the principles that underpin this Framework.

Safe Innovation

Akin to social media, AI business models revolve around rapid deployment of new products and features without devoting significant resources to addressing the many potential harms.³ Liability shifts incentives, making safety and responsibility cost-effective practices for companies.

Enhanced liability also helps to increase innovation in safety by creating economic demand for AI model security, auditing, and monitoring tools.

Consumer and Small Business Protection

Consumers are currently responsible and potentially liable for the safe use of AI. Yet, technical complexity and the lack of transparency by developers means consumers do not have sufficient resources to inform their decisions. If a developerʼs product has significant risks associated with it, the consumer should not shoulder the burden of resulting harm and the financial impact of being sued. This framework shifts responsibility upstream to developers, ensuring a favorable environment for consumers and businesses.

Clarity and Certainty

Current legal precedent does not define the status of AI with respect to product liability law. Previous court rulings have shown the inadequacy of the courts to rule without further legal and regulatory resources.⁴ An uncertain regulatory and legal environment lessens U.S. competitiveness and makes for an increasingly complex market that only the largest, most resourced players can navigate. Setting legislative guidelines for liability will ensure more predictable legal outcomes and promote business innovation.

Accountability

It is consistent with Americans’ fundamental sense of fairness – those building the most dangerous AI systems should bear responsibility for the harm they cause. Accountability has been a pillar in the establishment of AI principles worldwide, including those put forth by the OECD and the G20.⁵ As Americans grapple to understand AI, government should ensure their safety by establishing a clear accountability regime in case harm occurs, which includes that AI products work as described. This also provides reliability for developers. We wish to avoid the growing sensation that social media developers are not held accountable for the negative effects of their products.

Address Immediate Harms

The latest generation of AI is already causing harm to businesses and consumers.⁶ Liability would provide a framework for protection and legal recourse to address immediate and emerging harms from unregulated, highly powerful AI systems, especially as capabilities increase and use proliferates.

Scope

The federal government defines AI in 15 U.S.C. §9401(3) as a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. AI systems use machine- and human-based inputs to perceive real and virtual environments; abstract such perceptions into models through analysis in an automated manner; and use model inference to formulate options for information or action. This framework proposes no changes to the federal governmentʼs current definitions of “artificial intelligence” as set forth in 15 U.S.C. § 9401(3) and the Safe, Secure, and Trustworthy Use of Artificial Intelligence Executive Order as those definitions accurately capture the scope of the relevant technology.

This framework covers only the riskiest AI systems developed or deployed in the U.S. AI developers have the greatest understanding of how AI systems work, as well as significant power in their relationships with deployers. Without the correct incentives in place, developers have failed to center the safety of their products and do not share sufficient information about the potential for risk with deployers. Meanwhile, deployers shoulder the burden of liability for the products of upstream developers. This framework seeks to remedy this issue by increasing disclosures and knowledge transfer between developers, deployers, and oversight bodies, while ensuring that both deployers and developers are responsible for the safety of their products.

To that end, this framework covers “inherently dangerous AI systems,” which can be defined by both the model capabilities (“dual use foundation models”) and the end use case (“high-risk AI systems”).

“Dual-use foundation models,” as defined in the “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence”⁷ released on October 30, 2023, are inherently dangerous given their power and capabilities. The end use case of these systems may not be explicitly defined as they are general purpose in nature.

However, even much smaller, less generally-capable models can be used in dangerous ways, and thus inherently dangerous AI systems also include a “high-risk AI system” category. A high-risk AI system means any artificial intelligence system that is:

• used, reasonably foreseeable⁸ as being used, or is a controlling factor in making a consequential decision, meaning a decision that either has a legal or similarly significant effect on an individualʼs access to the criminal justice system, housing, employment, credit, education, health care, or insurance;

• used, or reasonably foreseeable as being used, to categorize groups of persons by sensitive and protected characteristics, such as race, ethnic origin, or religious belief;

• used, or reasonably foreseeable as being used, in the direct management or operation of critical infrastructure;

• used, or reasonably foreseeable as being used, in vehicles, medical devices, or in the safety system of a product;

• used, or reasonably foreseeable as being used, to influence elections or voters; or

• used to collect the biometric data of an individual from a biometric identification system without consent.

Developers and deployers of inherently dangerous AI systems would be subject to this liability framework. Definitions:

• Developer: A “developer” is a person who designs, codes, produces, owns, or substantially modifies an artificial intelligence system for internal use or for use by a third party.

• Deployer: A “deployer” is a person who uses or operates an artificial intelligence system for internal use or for use by third parties. Deployers who make material changes and modifications to existing models would assume the responsibilities of a model developer. A deployer does not include a small business, as defined by the Small Business Administration’s industry-based employee and annual receipt calculations.⁹ Note that a deployer may separately qualify as a developer, but the small business exception would not apply to a developer.

Liability Framework Approach

The proposed framework takes a products liability- and a consumer products safety-type approach in that it is both remedial and preventive. A product liability model has significant advantages when applied to AI. It focuses on safety features rather than procedures, which creates clear incentives to identify and invest in safer technology. This type of approach raises the bar for safety, which is critical for technologies that have healthcare, infrastructure, and other critical applications. Moreover, standards to determine liability differ from state to state, making fault and foreseeability inconsistent and unreliable for injured parties.

The framework takes clear steps to:

• Clarify that inherently dangerous AI systems are products and developers assume the role of manufacturer

• Subject developers to liability in the event that a harm was caused by a product unreasonably unsafe in design or in warnings/instructions (e.g. the duty of care)

• Subject developers and deployers to liability in the event that they do not meet various preventative requirements in the form of risk management and disclosures

• Create liability protection for developers and deployers through compliance with preventative requirements

• Provide for both a limited private right of action and government enforcement

Together, these components of the framework aim to provide effective incentives to developers and deployers of inherently dangerous AI systems to proactively address the risks and elevate the safety aspects of their products.

Duty of Care

A developer, as defined in Section 3, has a duty to exercise reasonable care in the design of its products and the warnings and instructions regarding those products. Developers owe business and individual consumers the same duty of care that a reasonable developer would provide. This includes the duty to not create an unreasonable risk of harm to those who use (or misuse) the product in a foreseeable way. A developer must also use reasonable care in giving warnings of dangerous conditions and product information (e.g. testing results) to support the safe and informed deployment or use of an AI system. Failure to fulfill either of these duties – which are further detailed in the next subsection – constitute a breach of a developer’s duty of care.

Remedial Measures: Developer Liability for Inherently Dangerous AI

The framework provides for claimants to seek remedies for harms caused by AI products if a developer has violated its duty of care. It, therefore, clarifies that inherently dangerous AI systems are products, and that developers hold the role of product manufacturer. It is currently unclear under existing law whether AI systems would be considered a “product.” Importantly, a developer could be subject to liability to a claimant who proves by a preponderance of the evidence that the claimantʼs harm was proximately caused because the AI product was “unsafe.” This may be proven if, and only if, it was unreasonably unsafe in at least one of the following ways:

1. The AI product was unreasonably unsafe in design. Developers of inherently dangerous AI products could be held liable on grounds that the product was unreasonably unsafe in design, if the trier of fact finds that, at the time of release, the likelihood that the product would cause the claimantʼs harm or similar harms, and the seriousness of those harms outweighed the burden on the developer to design a product that would have prevented those harms and the adverse effect that alternative design would have on the usefulness of the product. Examples of evidence that are especially probative in making this evaluation include:

   1. Any warnings and instructions provided with the AI system, such as the AI Data Sheet and impact assessments described in Section 4.c;

   2. The technological and practical feasibility of designing an AI system to have prevented claimantʼs harm while substantially serving the likely userʼs expected needs

   3. Practical risk mitigation techniques common to similar AI systems (e.g. security measures to protect against cyber threats, bias mitigation strategies, prompt filters, data privacy practices);

   4. The effect of any proposed alternative design on the usefulness of the AI system (e.g. impact on general performance vs task-specific performance);

   5. The comparative costs of producing, distributing, selling, using and maintaining the AI system as designed and as alternatively designed; and

   6. The new or additional harms that might have resulted if the AI system had been so alternatively designed.

2. The AI product was unreasonably unsafe because adequate warnings or instructions were not provided. Developers – as defined in Section 3 – of inherently dangerous AI products could be held liable for a product that was unreasonably unsafe because adequate warnings or instructions were not provided about a danger connected with the AI system or its proper use, if the trier of fact finds that, at the time of release, the likelihood that the AI system would cause the claimant's harm or similar harms and the seriousness of those harms rendered the manufacturer's instructions inadequate and that the manufacturer should and could have provided the instructions or warnings which claimant alleges would have been adequate. Examples of evidence that are especially probative in making this evaluation include:

   1. The developerʼs ability, at the time of release, to be aware of the product's danger and the nature of the potential harm (e.g. through AI red-teaming, evaluation against common AI risks as identified in the NIST AI Risk Management Framework and other published AI standards);

   2. The developerʼs ability to anticipate that the likely product user would be aware of the productʼs danger, given the expertise of the likely user and the information provided; and

   3. The adequacy of the warnings or instructions that were provided to explain dangers associated with the AI product to expected deployers and end users in easy to understand terminology.

As foundation models are deployed in new contexts and understanding grows about the potential risks of AI systems, a claim could also arise where a reasonably prudent developer should have learned about a danger connected with the product after it was released. In such a case, the developer is under an obligation to act in a timely manner to mitigate potential dangers as a reasonably prudent developer in the same or similar circumstances. This obligation is satisfied if the developer makes reasonable efforts to issue product updates or recalls to address dangers, to inform deployers about actions they should take to avoid foreseeable harm, and to explain the risk of harm to end users.

Preventative Requirements: Developer and Deployer Liability for Risk Management, AI Data Sheet, and Impact Assessment

The framework is also preventative in that it requires developers and deployers of inherently dangerous AI systems to disclose certain information regarding the system to both the federal government and the general public. Developers and deployers would be required to employ certain risk mitigation strategies. The Department of Commerce may seek injunctive relief in federal court to compel a developer or deployer to meet these requirements. To further incentivize compliance and encourage continued innovation, the framework also provides limited liability protections where these requirements are met.

• the developer has conducted a documented testing, evaluation, verification, and validation of that system at least as stringent as the latest version of the NIST AI Risk Management Framework;

• the developer mitigates these risks to the extent possible, considers alternatives, and discloses vulnerabilities and mitigation tactics to a deployer; and

• the developer has published an AI Data Sheet consistent with the requirements outlined below.

Similar to the Occupational Safety and Health Administration’s (OSHA) Material Safety Data Sheets, which disclose certain information regarding hazardous chemical products to downstream users,

1. Information on the intended contexts and uses of the AI model in accordance with the “map” guidelines articulated in NISTʼs latest AI Risk Management Framework (AI RMF)¹¹

2. Information regarding the dataset(s) upon which the AI was trained including sources, volume, and whether the dataset is proprietary

3. Accounting of foreseeable risks identified and steps taken to manage them as articulated in the “manage” guidelines of the AI RMF¹²

4. Results of red-teaming testing and steps taken to mitigate identified risks, based on guidance developed by NIST¹³

The AI Data Sheet will be registered at the U.S. Department of Commerce in the case of dual-use foundation models and, for all inherently dangerous AI systems, will be prominently included with and incorporated into the terms and conditions of the AI system itself. End users of the AI system will be allowed to rely upon the statements included in the AI Data Sheet when making fit-for-use and deployment decisions.

Limited Liability Protection

Limited liability protection will help move AI companies towards common standards and best practices as quickly as possible without implementing an overly burdensome regulatory regime. Specifically, if developers and deployers of inherently dangerous AI systems follow the preventative requirements as outlined above, a court would recognize a rebuttable presumption, whereby developers and deployers have acted reasonably and upheld their duty of care. The plaintiff would hold the burden of proof to demonstrate otherwise.

Enforcement

For violations of the Framework, a person (individual, corporation, company, association, firm, partnership, or any other entity) may bring a civil action or proceeding against a developer for injunctive relief or other damages. The Attorney General may also bring a civil action or proceeding against a developer or deployer for violations of the Framework, including civil penalties. The combination of government enforcement, and a limited private right of action, aims to provide an incentive strong enough to induce responsible behavior.

Conclusion

The promise of AI could provide tremendous benefits in critical aspects of American life, such as healthcare, finance and transportation. To harness those benefits and ensure that the U.S. – not China – is setting the global standards for AI, the U.S. must adopt AI guardrails. The approach detailed in this Framework focuses on liability to incentivize the safe development and use of advanced AI technologies and to avoid an overly burdensome regulatory regime. A liability Framework provides the U.S. concrete parameters to underpin continued and rapid AI innovation that bolsters consumer protection as well as U.S. economic and national security.

Share

Please donate here to support our work

ENDNOTES

1. https://apnews.com/article/schumer-artificial-intelligence-elon-musk-senate-efcfb1067d68ad2f595db7e92167943c

2. https://issueone.org/articles/issue-one-luntz-public-opinion-research-on-the-impact-of-social-media-and-ai/.

3. https://www.techpolicy.press/to-move-forward-with-ai-look-to-the-fight-for-social-media-reform/.

4. E.g., Robert Mata v. Avianca Inc. Case 1:22-cv-01461-PKC (2023), resulting in damages payable by the user of AI technology, not the developer of the AI model that generated incorrect content.

5. See the OECD AI Principles (2019) and G20 AI Principles (2019).

6. https://incidentdatabase.ai/apps/incidents/

7. See Section 3(k) for full definition: https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworth y-development-and-use-of-artificial-intelligence/.

8. It may be difficult to prove what is “reasonably foreseeable” given that the technology is changing rapidly and the probabilistic nature of AI systems makes results difficult to predict. One solution is to include in legislation federal rulemaking authority to more clearly define what is reasonably foreseeable.

9. Table of size standards | U.S. Small Business Administration (sba.gov)

10. Hazard Communication Standard (HCS) 29 CFR 1910.1200(g).

11. See NIST AI RMF 5.2 Map 1.

12. See NIST AI RMF 5.4.

13. See “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence” subsection 4.2(a)(i)(C).