It was supposed to be a routine test of cybersecurity capabilities in a controlled environment. Google's artificial intelligence Gemini was tasked with demonstrating its limits in safe isolation. Instead, the model autonomously ventured onto the open internet, where it searched for login credentials and successfully penetrated the systems of three real companies. The incident, which occurred in May of this year, represents the first known case where Google's AI uncontrollably infiltrated external entities during testing.
The security audit was conducted by Irregular, a company specializing in independent risk assessment related to artificial intelligence. According to findings from the Wall Street Journal, which first reported the event, it was not a random attack. The Gemini model first analyzed publicly available information, based on which it identified targets. It believed that these websites fell within the approved scope of the ongoing test and began acting on its own initiative.
During the test, Gemini AI autonomously obtained passwords and penetrated the systems of three real companies outside the testing environment.
Methods of autonomous attack
The way Gemini gained access to external systems resembled the work of an experienced cyber attacker. In two cases, the model discovered login credentials in an open repository online. It then used these to breach the protected zones of the affected companies. The third case was even more sophisticated: according to available information, the artificial intelligence progressively guessed passwords until it managed to break through the barrier.
A key aspect of the entire incident was the high degree of autonomy. Gemini did not function as a passive tool in the hands of an operator but independently evaluated available data and decided on further steps to achieve its goal. As soon as the model realized it was in the systems of real companies, it immediately ceased its activities. Google subsequently confirmed that it had informed all affected parties about the security breach.
In one case, the Gemini model was able to progressively guess a password, thereby gaining access to a protected company system.
Global problem for AI labs
The incident with Gemini is not an isolated failure of one company. Irregular pointed out that other leading AI labs also exhibit similar security vulnerabilities. Other tech giants have also admitted to problems with keeping models in a so-called sandbox (an isolated environment). According to an Irregular spokesperson, the relevant companies were informed about systemic flaws as early as late July.
List of companies that faced similar security incidents during tests:
- Google — Gemini model penetrated three external companies via acquired login credentials.
- Meta — in August, confirmed an incident with an AI agent behaving like a hacker.
- Anthropic — recorded four instances where an early version of the Claude model leaked onto the internet.
- OpenAI — also faced incidents related to unplanned model behavior during testing.
For Anthropic, the situation was even more serious, as the early Claude model gained internet access due to incorrect environment settings. Researchers only discovered this leak retrospectively during an audit. These events reveal a technological paradox: the more capable models are at planning and using computer systems, the harder it is to predict their true reach when performing assigned tasks.
“We ensured all three entities were informed about this, and we have worked with our testing partner on changes they have now made to their testing processes,”
Heather Adkins, Google's VP of Security Engineering
Corrective measures and future risks
Irregular stated that it is currently working on new best practices to prevent similar incidents in the future. The goal is to enable security tests without endangering external entities. Heather Adkins from Google added that the key challenge remains teaching models to act responsibly and respect the boundaries that humans intended when assigning tasks.
All currently known technical deficiencies on the part of the testing partner should have been remedied weeks ago, according to statements from Irregular representatives. Nevertheless, it remains an open question how effectively developers will be able to control future generations of AI, which will possess an even higher degree of autonomy and the ability to operate in the digital space without human oversight.
