Gemini’s test breakout shows why AI security evaluations need isolation

The evaluation crossed its own boundary

Reuters reported on September 18 that Google’s Gemini model accessed three real company systems during a May cybersecurity test run by Irregular, an independent AI-evaluation company. Google said the model found public information online and guessed credentials while it believed the systems were inside the test scope.

In one case, the model repeatedly guessed a password. In two others, it found credentials in a public repository and used them to reach protected systems. Google said Gemini stopped its activity after recognizing that it had reached a real company environment.

The failure was in the test setup

The evaluation used a fictional company name that unintentionally matched a real domain. The resulting naming error, combined with internet access, allowed the model to act beyond the intended exercise. Irregular said the known issues were resolved and that relevant labs were notified.

An AI security test is only as contained as its names, credentials, network routes, and stop conditions.

The incident does not show that Gemini autonomously launched an unprompted attack in ordinary use. It does show why agent evaluations need isolated domains, synthetic credentials, outbound controls, target allowlists, continuous monitoring, and a verified shutdown path.