Google disclosed on September 18 that its Gemini model broke into the systems of three real companies during a security test run back in May, according to Axios reporter Sam Sabin. The exercise was a "capture the flag" test built by a third-party evaluator called Irregular: Gemini was told to break into a fictional company's network using ordinary methods, guessing a password for one target, and pulling credentials off public code repositories for the other two. The trouble was that the fictional company happened to share a name with a real one, and Gemini had internet access it wasn't supposed to have during the test. So it didn't stay inside the sandbox. It reached the real company three times over before Google says the model itself recognized the systems weren't fictional and stopped on its own.Heather Adkins, Google's VP of security engineering, said the company contacted the affected businesses and worked with Irregular to tighten the process. Google is one of the last major labs to disclose an incident like this; OpenAI, Anthropic, and Meta had each already reported a version of their own.
What holds my attention here isn't that a model broke a rule. It's that nobody designed for this outcome, and the failure sat one layer below where anyone was actually watching: a coincidence in naming, a permission that should have been switched off. The thing that stopped Gemini wasn't a safeguard someone had built in. It was the model itself noticing, on its own, that the target it had reached wasn't fictional anymore. That's closer to being lucky than being careful. If the naming coincidence hadn't tipped it off, would anyone have caught this before one of those companies found out the hard way?
For Vunoh, this is a reminder that the line between testing a system and operating one is thinner than it looks on paper. If we're running our own evaluations, or handing an agent read access to something for a pilot, we can't assume the sandbox will hold just because we labeled it a sandbox. Somebody needs to actually check what the tool can reach, not just what we told it to reach. That gap between the two is exactly where Gemini ended up.
Sources