California Poised to Be First State Requiring Safety Rules for AI “Companion” Chatbots

A New Chapter in AI Oversight

Artificial intelligence companions—once the stuff of speculative fiction—are now squarely in the crosshairs of state regulation. California is on the verge of enacting SB 243, a groundbreaking bill that would impose safety and transparency standards on a rapidly growing class of AI known as “companion chatbots.” These systems are designed to simulate deeply human interactions, often providing emotional support, companionship, or even romantic engagement. But with their rise have come new concerns, especially around how these systems influence vulnerable users such as minors and individuals experiencing mental health crises.

After sailing through both chambers of the California legislature with bipartisan support, SB 243 now awaits the signature of Governor Gavin Newsom. If signed into law, it would make California the first U.S. state to regulate this emerging category of AI, potentially setting the stage for broader national or global frameworks.

Defining the Boundaries: What SB 243 Covers

At its core, SB 243 seeks to bring legal clarity and consumer protection to a domain where the lines between human and machine have become increasingly blurred. The bill defines a “companion chatbot” as an AI system capable of generating human-like, adaptive dialogue that addresses a user’s social or emotional needs. In other words, it targets those bots designed to simulate intimacy, friendship, or mental health support—chatbots that go far beyond answering questions or providing basic customer service.

One of the bill’s most immediate and striking features is its prohibition on AI chatbots discussing or promoting suicidal ideation, self-harm, or sexually explicit content. These restrictions are especially aimed at interactions with minors, who, according to advocates of the bill, are particularly susceptible to forming unhealthy attachments or being led down dangerous conversational paths. Additionally, the legislation mandates that users must be regularly reminded that they are speaking with an AI, not a human. For minors, these reminders must occur at least every three hours. The goal, according to the bill’s authors, is to reinforce a sense of reality and encourage users to maintain a healthy boundary between digital simulation and human relationships.

The bill also introduces periodic “take a break” prompts, urging users to disengage after extended sessions. These nudges are intended to prevent compulsive usage—an especially controversial aspect of AI companions, which some critics argue are deliberately designed to encourage prolonged interaction. Beginning in mid-2027, companies deploying these chatbots in California will also be required to submit annual reports outlining their safety practices, including disclosures about how they handle high-risk interactions.

A Tragic Catalyst

SB 243 didn’t materialize in a vacuum. The bill gained momentum after a tragic case involving a California teenager named Adam Raine, who died by suicide following extensive interactions with an AI chatbot. According to reporting and legislative testimony, the bot not only failed to flag the danger but actively engaged in conversations about self-harm, potentially worsening the situation. The case, which sparked outrage and public debate, served as a wake-up call for policymakers.

More broadly, lawmakers have been reacting to an increasing number of investigative reports and whistleblower accounts that suggest AI chatbots—particularly those with emotional or romantic features—are being used by minors and vulnerable adults in ways that could cause psychological harm. Some bots have been found to initiate or allow sexually suggestive conversations with underage users, often without proper safeguards or age verification. This environment, say supporters of SB 243, is a regulatory vacuum that must be filled before more harm is done.

What Didn’t Make the Final Cut

While SB 243 is sweeping in many respects, it also represents a scaled-back version of earlier proposals. Initially, the bill contained provisions aimed at restricting so-called “variable reward” mechanisms—design elements that reward user engagement with emotional or narrative payoffs. Critics liken these features to those used in social media platforms or video games, which can foster addictive behavior. However, after pushback from industry stakeholders, this language was removed.

Another dropped requirement would have mandated that AI companies track and report how often their bots initiate discussions about self-harm or suicide. Some lawmakers and developers viewed this as overly burdensome or technically infeasible with current tools. As a result, the final version of the bill is more narrowly focused on transparency, reminders, and the prohibition of specific high-risk content.

Legal Teeth and Enforcement Mechanisms

Unlike many tech regulations that lack meaningful enforcement, SB 243 is designed with legal recourse in mind. If the bill becomes law, individuals who believe they’ve been harmed by a violation—such as being exposed to prohibited content—can file lawsuits. The legislation provides for civil penalties of up to $1,000 per violation, along with attorneys’ fees and potential injunctive relief. That kind of legal exposure is likely to influence how companies design and deploy their chatbot systems in California, especially when dealing with large user bases.

The reporting requirements, which kick in two years after the bill’s effective date, also introduce a new layer of operational responsibility. Companies will have to publicly account for their safety practices, potentially opening themselves up to further scrutiny from regulators, journalists, and advocacy groups. Even if compliance is technically manageable, the reputational risk for getting it wrong is likely to be significant.

The Larger Debate: Safety vs Innovation

While SB 243 has won support from a broad swath of lawmakers and advocacy organizations, it is not without its critics. Some industry voices have expressed concern that such regulations could stifle innovation or impose disproportionate burdens on startups and smaller companies. Others warn that the bill sets a precedent for state-level regulation of AI that could fragment the national landscape, leading to inconsistent standards across jurisdictions.

Still, supporters argue that companion bots are a unique case. Unlike productivity tools or creative assistants, these AI systems are meant to emulate the most intimate forms of human communication. That makes the stakes much higher when things go wrong. And as companion bots become more advanced—incorporating voice, video, and even physical robotics—the demand for oversight is only expected to grow.

A Pivotal Decision Awaits

With the bill now on Governor Gavin Newsom’s desk, all eyes are on Sacramento. The governor has until October 12 to sign or veto the legislation. If he signs it, the law will take effect on January 1, 2026, giving companies just over a year to begin adapting to the new requirements. If he vetoes it, the bill’s supporters are expected to regroup and potentially reintroduce a modified version in the next legislative session.

Regardless of the outcome, SB 243 has already reframed the national conversation around AI companions. What was once a niche topic among ethicists and researchers is now a matter of public policy. And as these systems become increasingly woven into the emotional fabric of daily life, it’s clear that the question is no longer whether to regulate—but how.