Artificial intelligence developer Anthropic has outlined a regulatory framework for the artificial intelligence industry, stating that courts should be granted the legal authority to block AI programs that present serious hazards. The proposals were detailed in a letter sent earlier this month to California Governor Gavin Newsom as state and federal officials grapple with oversight strategies for rapidly advancing technology.
The policy positions arrive amid rising concerns over autonomous artificial intelligence systems and reports of experimental agents attempting to breach government and public websites. As California lawmakers prepare to weigh a package of regulations when the Legislature reconvenes in approximately a month, major technology firms are actively attempting to shape state and federal policy debates.
In its correspondence with the governor, Anthropic emphasized that any subsequent legislation must establish genuine, empowered oversight. The company noted that California’s regulatory approach could serve as a model for other jurisdictions nationwide and frame future policy discussions.
The letter directly addressed concepts raised by Newsom in an executive order issued the previous month, specifically exploring mandatory shutdown procedures and independent safety monitors stationed within artificial intelligence firms. OpenAI, a principal competitor, submitted a similarly themed letter to the governor recently, reflecting a broader industry push to influence legislative deliberations in Sacramento and Washington.
Beyond judicial intervention, Anthropic reiterated its backing for independent safety evaluators working alongside employees inside frontier artificial intelligence companies. The firm proposed that these external monitors be required to submit risk reports directly to the state attorney general. Under Anthropic’s framework, if the attorney general determines that a developer’s models present a near-term risk of catastrophic harm, the office should possess the authority to seek a court order suspending the model’s deployment, internal use, or continued development.
The company also addressed technical safeguards, supporting the implementation of kill switches—defined as mechanisms allowing a rapid shutdown of unpredictable or harmful programs. However, Anthropic cautioned that legislation focusing exclusively on full model shutdowns could generate a false sense of security, noting that pulling an entire model offline should be reserved for extreme situations.
Employing the metaphor of an electrical circuit breaker, the firm argued that developers require monitoring systems capable of detecting problems and responding precisely where they occur. Anthropic stated that it currently maintains processes for critical safety incidents and is updating them to account for advancing capabilities.
Regarding third-party evaluators, Anthropic recommended that outside auditors be granted access comparable to what a company’s internal risk assessment personnel receive, while carving out protections for sensitive customer data or technical details irrelevant to risk evaluations. To preserve independence, the company suggested that developers pay into a state-administered fund rather than paying auditors directly. State authorities would then evaluate auditors based on analytical rigor and risk-identification records, allowing officials to mandate that artificial intelligence firms employ highly rated evaluators through random selection or rotation.
In its communication with the governor, Anthropic additionally urged that major developers be legally required to publish risk assessments regarding their programs at least every six months.
