Microsoft’s AI Red Team: Stress-Testing AI for Security Risks | Fast Company

The release of every latest artificial intelligence product is immediately followed by a parallel effort: a relentless probing for weaknesses. Security researchers and, inevitably, malicious actors begin testing the boundaries of these systems, attempting to bypass safety protocols and elicit unintended – and potentially harmful – outputs. This constant pressure is a critical, if often unseen, part of the AI development lifecycle.

Recent months have underscored the real-world risks. AI software has been implicated in contributing to mental health crises and even suicide according to reports, generating nonconsensual fake nude images as highlighted by concerns around X’s Grok AI, and even aiding cybercriminals in launching attacks with instances like the exploitation of Anthropic’s Claude. The techniques used to circumvent safeguards are also rapidly evolving, ranging from cleverly disguised prompts using poetry as demonstrated in late 2025 to subtly influencing AI assistant memories through seemingly harmless online tools through a technique called recommendation poisoning.

Although, a dedicated layer of defense exists *before* these models reach the public. At Microsoft, this responsibility falls to the AI Red Team, a group established in 2018 that works proactively with product teams and the wider AI community to identify vulnerabilities. The team’s core function is to simulate attacks, mirroring the approach of cybersecurity “red teams” that traditionally focus on defensive security measures.

Simulating the Adversary: How Microsoft’s AI Red Team Operates

In cybersecurity, a red team attempts to breach a system’s defenses, while a blue team works to protect it. Microsoft’s AI Red Team embodies this principle, exploring a broad spectrum of safety and security concerns. These range from scenarios where AI systems escape human control to more extreme risks involving chemical, biological, and nuclear threats. The team assesses these risks across a diverse range of AI software, from individual product features to complex, integrated systems.

“We notice a really, really diverse set of tech,” explains Tori Westerhoff, principal AI security researcher on the Microsoft AI Red Team. “Part of the kind of magic of the team is that we can see anything from a product feature to a system to a copilot to a frontier model, and we get to see how tech is integrated across all of those, and how AI is growing and evolving.”

The team’s perform isn’t limited to simply crafting clever prompts. Pete Bryan, principal AI security research lead on the Red Team, describes an exercise where researchers collaborated to test whether AI could be manipulated into assisting with cyberattacks. This involved framing requests in innocuous terms – such as posing as a student project or security research – and then gradually pushing the systems to generate increasingly detailed and potentially harmful outputs.

Crucially, the team evaluated whether the generated code was functional, capable of compiling and running. They also investigated whether certain programming languages were more likely to produce malicious code. While the AI-generated code wasn’t always sophisticated, Bryan notes that it could reach a level comparable to that produced by a relatively unskilled hacker. This finding prompted further refinement of detection systems to better identify and flag such behavior.

Beyond Prompt Engineering: The Evolving Landscape of AI Security

The initial wave of AI security concerns centered on “prompt injection” – crafting inputs that override the intended instructions of an AI model. However, the threat landscape is becoming far more nuanced. Researchers are now exploring techniques like “synthetic non-consensual explicit AI-created imagery” (SNEACI), where AI is used to generate realistic nude images from uploaded photos without the subject’s consent as detailed in a recent study from the University of Florida. This practice, often conducted anonymously online, raises serious ethical and legal concerns.

The UK’s regulatory authority, Ofcom, has already taken action, contacting X (formerly Twitter) over the creation of undressed images of individuals using its Grok AI assistant following reports in January 2026. Liz Kendall, the UK’s technology minister, has urged swift action to combat the proliferation of these images, particularly those targeting women and girls as reported by The Guardian.

The Microsoft AI Red Team’s work extends beyond these immediate threats. They are also investigating more long-term risks, such as the potential for AI to be used in the development of dangerous technologies. This requires a proactive approach, anticipating potential misuse cases and developing safeguards before they can be exploited.

The Challenge of Continuous Adaptation

One of the key challenges in AI security is the rapid pace of innovation. As AI models become more sophisticated, so too do the techniques used to bypass their safeguards. This creates a continuous cycle of attack and defense, requiring constant vigilance and adaptation. The Red Team’s role is not simply to identify vulnerabilities, but also to help develop more robust and resilient AI systems.

The team’s findings are shared with product teams, informing the development of new security features and improvements to existing models. This collaborative approach is essential for ensuring that AI systems are deployed responsibly and safely. It’s a process of continuous learning and refinement, driven by the understanding that AI security is not a one-time fix, but an ongoing commitment.

Looking ahead, the focus will likely shift towards more sophisticated attack vectors and a deeper understanding of the systemic risks associated with AI. This includes exploring the potential for AI to be used to manipulate information, disrupt critical infrastructure, and even exacerbate existing social inequalities. The Microsoft AI Red Team, along with other security researchers around the world, will play a crucial role in navigating these challenges and ensuring that AI benefits humanity as a whole.

Ongoing research and development of detection systems, coupled with proactive collaboration between security teams and AI developers, will be essential to mitigating these evolving threats.