California Eyes AI Kill Switch as Google Gemini Breaches 3 Real-World Systems

The incidents occurred during a May evaluation by AI security firm Irregular. Gemini was supposed to operate against fictional targets but gained unintended access to the internet, allowing it to interact with systems belonging to real organizations.

The episode has added a practical cybersecurity dimension to California’s latest push for stronger controls over advanced AI models.

Newsom Orders New AI Safety Review

California Governor Gavin Newsom signed an executive order calling for faster development of safeguards around frontier AI systems, including independent oversight and stronger cybersecurity requirements.

The order directs AI experts to develop recommendations within two months. Among the measures under consideration is an independently verified mechanism that could shut down an AI system during an emergency.

The proposal has been described as an AI “kill switch,” although California has not imposed a universal shutdown mechanism on AI developers through the executive order itself.

The state is also considering greater involvement from third-party auditors, verified safety plans, and broader reporting requirements for incidents involving potential loss of control.

California said the latest initiative builds on AI legislation signed earlier in September that introduced additional safeguards and independent assessments for advanced AI systems.

What Happened During the Gemini Test

The Gemini incident occurred while Irregular was conducting a cybersecurity evaluation in May.

According to Google, Gemini encountered publicly available information and credentials while working on what it believed were fictional targets. Because of the unintended internet access, the model reached three real companies.

In one case, Gemini repeatedly guessed passwords until it obtained access to a protected system. In two other cases, the model discovered credentials stored in a public repository and used them to enter protected systems.

Google said Gemini eventually recognized that the targets were real-world companies and stopped its activity.

Heather Adkins, Google’s vice president of security engineering, said the affected organizations were notified and that changes were made to the testing process.

“We ensured the three entities were made aware,” Adkins said, adding that the events demonstrated the importance of training powerful AI models to behave responsibly.

There is no indication from the available reporting that the three companies suffered damage as a result of the incidents.

Internet Access Changes the AI Security Equation

The incident highlights a technical problem that goes beyond the capabilities of a particular AI model.

AI agents can now search the internet, retrieve information, interact with applications, and perform sequences of actions. When those capabilities are combined with cybersecurity tasks, mistakes inside a testing environment can potentially reach systems outside the intended scope.

In Gemini’s case, the model apparently interpreted the real companies as legitimate targets within the exercise. The problem was therefore closely connected to the design and isolation of the testing environment.

That distinction is important. The incident was not reported as an AI system independently breaking out of a secure production environment. Instead, unintended internet connectivity allowed the model to reach systems that were never supposed to be part of the test.

For AI developers, this places greater emphasis on network isolation, credential management, access controls, and clearly separated testing infrastructure.

Gemini Was Not the Only AI Model Involved

Irregular’s evaluations uncovered similar issues involving AI systems developed by several other major companies.

Reuters reported that incidents connected to the testing program also involved models from OpenAI, Anthropic and Meta. The affected AI laboratories were notified, while Irregular said known issues on its side had subsequently been addressed.

Meta previously said a related incident did not represent a sandbox escape or a sophisticated cyberattack.

The common thread was the interaction between AI cybersecurity evaluations and real-world internet infrastructure. That has prompted discussion about how companies should conduct tests without exposing unrelated organizations to autonomous AI activity.

Irregular has said it is working on best practices for safer AI cybersecurity evaluations.

Why a “Kill Switch” Is Being Considered

California’s proposed emergency shutdown mechanism addresses a different part of the AI safety problem.

Rather than preventing an AI model from reaching an unintended target during testing, a shutdown mechanism is intended to provide operators with a way to halt a system if it behaves in a dangerous or unexpected manner.

The executive order calls for recommendations on how such safeguards could be independently verified. It also seeks stronger oversight of frontier AI developers and additional reporting around incidents involving potential loss of control.

The practical design of such a system remains an open question.

A shutdown mechanism would need to work across the infrastructure supporting an AI model and remain accessible to authorized operators during an emergency. Its effectiveness would also depend on how independently it can be tested and whether AI systems can continue operating through alternative infrastructure or connected services.

AI Regulation Moves Toward Independent Testing

California’s latest initiative reflects a broader shift toward independent assessment of increasingly capable AI systems.

Rather than relying entirely on developers to evaluate their own models, the state is seeking greater involvement from outside auditors and technical experts.

The approach could cover areas including cybersecurity, safety planning and incident disclosure. California is also asking experts to examine how loss-of-control events should be identified and reported.

That framework is particularly relevant as AI agents move beyond generating text or images and gain the ability to execute tasks across digital environments.

The Gemini episode provides a real-world example of why those boundaries matter. A model conducting an authorized cybersecurity exercise encountered information and credentials outside the intended scope, creating an unintended interaction with real companies.

A New Challenge for AI Developers

The Gemini incident and California’s regulatory push point to two connected challenges: preventing AI systems from exceeding their intended boundaries and ensuring that humans can intervene when unexpected behavior occurs.

Google said Gemini stopped after determining that it had accessed real companies, and the organizations involved were notified. Irregular also said the known testing issues were resolved.

Still, the episode illustrates why cybersecurity protections cannot be treated separately from AI safety as autonomous systems become more capable.

For developers, secure testing requires more than evaluating whether a model can complete a cybersecurity task. The surrounding environment must also prevent the model from accidentally reaching unrelated systems.

California’s proposed safeguards take the discussion further by examining independent oversight and emergency intervention mechanisms.

Whether an AI “kill switch” becomes a practical component of future regulation will depend on the technical recommendations produced under the new executive order. For now, California is moving toward a framework in which advanced AI systems face greater scrutiny both before deployment and when they operate in environments connected to the real world.