Executives from leading companies developing advanced AI—including Google DeepMind, OpenAI, xAI, and Anthropic—have consistently highlighted the significant implications of their work. Each has, at various times, cautioned that powerful AI could lead to catastrophic outcomes for humanity. Despite these warnings, within an industry characterized by proprietary information and non-disclosure agreements, potential whistleblowers have been confronted with a difficult dilemma: remain silent, or come forward and face possible legal repercussions.

A groundbreaking California law seeks to simplify this choice. The Transparency in Frontier Artificial Intelligence Act (SB 53), enacted by Governor Gavin Newsom on September 29, provides the legal safeguard long sought by industry insiders, shielding those who report misconduct from retaliatory actions by their employers.

This legislation applies to any company operating within California that utilizes substantial computational power for training AI models, a criterion designed to regulate only cutting-edge developers while exempting smaller firms. It couples these new whistleblower protections with transparency mandates, requiring developers to publish risk mitigation strategies and disclose safety test results, thereby formalizing commitments that critics contend were previously unmet. Companies are also obliged to report “critical safety incidents” to the California Office of Emergency Services (OES).

The Rationale Behind Demands For Whistleblower Safeguards

Although “Godfather of AI” Geoffrey Hinton departed Google in 2023 to discuss the technology’s existential risks, specific calls for whistleblower protections did not gain prominence until May 2024. This followed a Vox report detailing OpenAI’s alleged use of “highly restrictive offboarding agreements” intended to stifle criticism. OpenAI employees, much like others in Silicon Valley, received a significant portion of their compensation in the form of equity. However, upon leaving the company, they were reportedly required to sign an agreement to retain their vested equity. This agreement supposedly contained clauses permanently forbidding them from criticizing their former employer and preventing them from even mentioning the agreement’s existence. This issue came to light after former employee Daniel Kokotajlo posted on an online forum about his refusal to sign, believing he had forfeited millions of dollars worth of vested equity.

The subsequent day, OpenAI CEO Sam Altman reportedly disavowed any knowledge of these clauses, stating, “This is my responsibility and one of the rare occasions I’ve felt truly ashamed leading OpenAI; I wasn’t aware this was occurring, and I should have been.” OpenAI later informed both current and former staff, through messages, that these provisions would no longer be enforced and that the objectionable language would be removed from all future exit paperwork. Yet, the credibility of Altman’s denial was questioned when, on May 22, documents surfaced that reportedly bore his signature—alongside those of other senior executives—explicitly authorizing the provisions. In June 2024, amidst the controversy surrounding OpenAI’s then-rescinded off-boarding agreements, Kokotajlo and 13 current and former employees from OpenAI and Google DeepMind advocated for enhanced whistleblower protections in an open letter.

Scope of Safeguards Under the New Legislation

In alignment with a primary recommendation from the open letter, California’s new legislation extends beyond conventional whistleblower protections, which typically cover only illegal activities. It forbids AI developers from enforcing policies that prevent employees responsible for risk assessment from reporting violations of the act or “catastrophic risks”—defined as posing a danger to 50 lives or causing $1 billion in damages—to the Attorney General, regulatory bodies, or their managers. Employees are not required to be certain of a catastrophic risk; possessing “reasonable cause” to believe it is sufficient for coverage under the new law.

Limitations of California’s Whistleblower Safeguards

Jacob Hilton, a former OpenAI researcher and one of the signatories of the open letter, considers the new protections “an important step forward.” Nevertheless, he expresses a desire for similar protections to be adopted at the federal level, referencing the bipartisan “AI Whistleblower Protection Act” introduced in Congress in May. Hilton observes that the “catastrophic risk” threshold means that lesser, though still significant, harms would not be covered. Another anonymous signatory of the open letter informed TIME that while addressing catastrophic risks is essential, “there could be other risk categories where additional protections would be desirable.” 

Lawrence Lessig, a professor of law at Harvard who provided pro bono legal advice to Kokotajlo and other signatories of the letter, asserts that extending beyond mere disclosures of illegal activity is a crucial provision. However, Lessig is concerned that ambiguity regarding what constitutes protected disclosures might deter employees from coming forward. He indicates he would have preferred a “good faith” standard, which focuses on an employee’s sincere conviction rather than the robustness of their evidence. As Lessig suggests, this would be a “far simpler benchmark for a technology professional to depend on” than requiring them to function as a legal or risk expert. Under the new legislation, this more lenient “good faith” standard is exclusively reserved for an internal, anonymous reporting channel within large AI corporations.

Lessig states that the hazards faced by past whistleblowers underscore the necessity of such protections. He concludes, “When individuals within an organization are prepared to forgo $2 million merely for the liberty to speak, we should be eager to hear what they have to convey.”