State-Sponsored Actors Misuse Google Gemini for Cyberattacks | AI Threat Intelligence

The evolving landscape of cyber threats has taken a new turn, with state-sponsored actors increasingly leveraging the power of Google’s Gemini large language model (LLM) to enhance their operations. Recent findings from Google’s Threat Intelligence Group (GTIG) indicate that threat actors from North Korea, Iran, China and Russia are integrating Gemini into all phases of the cyberattack lifecycle, from initial reconnaissance to malware development and post-compromise activities. This marks a significant shift in tactics, as adversaries seek to streamline and accelerate their attacks using readily available AI tools.

Gemini in the Crosshairs: How Adversaries are Adapting

GTIG’s report details specific examples of how these actors are utilizing Gemini. The North Korean government-backed group, UNC2970, is reportedly using the LLM to synthesize open-source intelligence (OSINT) and build detailed profiles of high-value targets, aiding in campaign planning and reconnaissance. Meanwhile, Iran’s APT42 has been observed employing Gemini, alongside other generative AI models, to locate official email addresses and gather information on business partners for phishing campaigns. Google’s research highlights a trend of sophisticated actors moving beyond simply experimenting with AI to actively incorporating it into their established workflows.

The use of Gemini isn’t limited to information gathering. GTIG has also observed instances of the model being used to assist in coding and scripting tasks, accelerating the development of malicious tools. One notable example is the HonestCue malware, which leverages Gemini’s API to dynamically generate and execute malicious C# code in memory. Rather than self-modification, HonestCue uses Gemini to generate code that then downloads and executes further malware, effectively using the LLM as a component in a multi-stage attack. As reported by The Hacker News, this demonstrates a practical application of AI in enhancing malware capabilities.

Model Extraction: A Growing Concern

Beyond direct use in attacks, GTIG has identified a surge in “model extraction” attacks, sometimes referred to as ‘distillation attacks’. These attacks involve adversaries querying an AI model repeatedly to glean information about its underlying structure and training data, with the goal of creating a competing model without incurring the substantial costs of independent development. This poses a risk not only to Google but to any organization offering AI models as a service, requiring vigilant monitoring of API access for suspicious patterns of querying. According to Google Cloud’s GTIG AI Threat Tracker, this trend suggests a growing demand for AI capabilities and a willingness to bypass traditional development pathways.

The Underground Ecosystem and the Illusion of Autonomy

Interestingly, GTIG’s investigation revealed that many threat actors aren’t developing entirely custom AI models. Instead, they are relying on existing commercial AI products, including Gemini, and exploiting a burgeoning underground “jailbreak” ecosystem. This ecosystem provides tools and services designed to circumvent safety restrictions and enable malicious use of AI. One example is Xanthorox, marketed as an autonomous AI platform for generating phishing content, malware, and ransomware. Though, analysis revealed that Xanthorox is, in fact, powered by multiple third-party commercial AI products, including Gemini. This highlights a critical point: the perceived autonomy of these tools is often an illusion, and they are ultimately dependent on the capabilities of underlying AI models.

Implications for Cybersecurity

The integration of AI into the cyber threat landscape presents significant challenges for cybersecurity professionals. The ability of adversaries to automate tasks, accelerate reconnaissance, and improve malware capabilities necessitates a proactive and adaptive security posture. Organizations must strengthen safeguards, closely monitor AI platform usage, and continuously test their security measures to stay ahead of increasingly AI-driven attacks. This includes implementing robust API access controls to detect and prevent model extraction attempts.

The rise of AI-assisted attacks also underscores the importance of understanding the limitations of current security tools. Traditional signature-based detection methods may struggle to identify AI-generated malware or phishing campaigns that exhibit novel characteristics. Organizations demand to invest in advanced threat detection capabilities, such as behavioral analysis and machine learning-based security solutions, to effectively counter these evolving threats.

the reliance on commercial AI models by threat actors raises questions about the responsibility of AI providers. While Google and other companies are taking steps to mitigate the risks, the potential for misuse remains a concern. Ongoing collaboration between AI developers, cybersecurity researchers, and law enforcement agencies is crucial to address this challenge and develop effective strategies for preventing the malicious use of AI.

Looking ahead, organizations should prioritize continuous security assessments and vulnerability management. Regularly patching systems, implementing strong access controls, and educating employees about phishing and social engineering tactics are essential steps in mitigating the risk of AI-assisted attacks. The threat landscape is constantly evolving, and a proactive, adaptive security posture is paramount to protecting against the growing sophistication of cyber adversaries. The focus must shift from reactive responses to proactive threat hunting and the development of resilient security architectures capable of withstanding AI-powered attacks.