-
Lessons From Red Teaming 100 Generative AI Products
Authors:
Blake Bullwinkel,
Amanda Minnich,
Shiven Chawla,
Gary Lopez,
Martin Pouliot,
Whitney Maxwell,
Joris de Gruyter,
Katherine Pratt,
Saphir Qi,
Nina Chikanov,
Roman Lutz,
Raja Sekhar Rao Dheekonda,
Bolor-Erdene Jagdagdorj,
Eugenia Kim,
Justin Song,
Keegan Hines,
Daniel Jones,
Giorgio Severi,
Richard Lundeen,
Sam Vaughan,
Victoria Westerhoff,
Pete Bryan,
Ram Shankar Siva Kumar,
Yonatan Zunger,
Chang Kawaguchi
, et al. (1 additional authors not shown)
Abstract:
In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have lea…
▽ More
In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned:
1. Understand what the system can do and where it is applied
2. You don't have to compute gradients to break an AI system
3. AI red teaming is not safety benchmarking
4. Automation can help cover more of the risk landscape
5. The human element of AI red teaming is crucial
6. Responsible AI harms are pervasive but difficult to measure
7. LLMs amplify existing security risks and introduce new ones
8. The work of securing AI systems will never be complete
By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.
△ Less
Submitted 13 January, 2025;
originally announced January 2025.
-
PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System
Authors:
Gary D. Lopez Munoz,
Amanda J. Minnich,
Roman Lutz,
Richard Lundeen,
Raja Sekhar Rao Dheekonda,
Nina Chikanov,
Bolor-Erdene Jagdagdorj,
Martin Pouliot,
Shiven Chawla,
Whitney Maxwell,
Blake Bullwinkel,
Katherine Pratt,
Joris de Gruyter,
Charlotte Siska,
Pete Bryan,
Tori Westerhoff,
Chang Kawaguchi,
Christian Seifert,
Ram Shankar Siva Kumar,
Yonatan Zunger
Abstract:
Generative Artificial Intelligence (GenAI) is becoming ubiquitous in our daily lives. The increase in computational power and data availability has led to a proliferation of both single- and multi-modal models. As the GenAI ecosystem matures, the need for extensible and model-agnostic risk identification frameworks is growing. To meet this need, we introduce the Python Risk Identification Toolkit…
▽ More
Generative Artificial Intelligence (GenAI) is becoming ubiquitous in our daily lives. The increase in computational power and data availability has led to a proliferation of both single- and multi-modal models. As the GenAI ecosystem matures, the need for extensible and model-agnostic risk identification frameworks is growing. To meet this need, we introduce the Python Risk Identification Toolkit (PyRIT), an open-source framework designed to enhance red teaming efforts in GenAI systems. PyRIT is a model- and platform-agnostic tool that enables red teamers to probe for and identify novel harms, risks, and jailbreaks in multimodal generative AI models. Its composable architecture facilitates the reuse of core building blocks and allows for extensibility to future models and modalities. This paper details the challenges specific to red teaming generative AI systems, the development and features of PyRIT, and its practical applications in real-world scenarios.
△ Less
Submitted 1 October, 2024;
originally announced October 2024.
-
Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle
Authors:
Emman Haider,
Daniel Perez-Becker,
Thomas Portet,
Piyush Madan,
Amit Garg,
Atabak Ashfaq,
David Majercak,
Wen Wen,
Dongwoo Kim,
Ziyi Yang,
Jianwen Zhang,
Hiteshi Sharma,
Blake Bullwinkel,
Martin Pouliot,
Amanda Minnich,
Shiven Chawla,
Solianna Herrera,
Shahed Warreth,
Maggie Engler,
Gary Lopez,
Nina Chikanov,
Raja Sekhar Rao Dheekonda,
Bolor-Erdene Jagdagdorj,
Roman Lutz,
Richard Lundeen
, et al. (6 additional authors not shown)
Abstract:
Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3…
▽ More
Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3 series of language models. We utilized a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to cover a variety of harm areas in both single and multi-turn scenarios. Our results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks. Finally, we include additional red teaming strategies and evaluations that were used to test the safety behavior of Phi-3.5-mini and Phi-3.5-MoE, which were optimized for multilingual capabilities.
△ Less
Submitted 22 August, 2024; v1 submitted 18 July, 2024;
originally announced July 2024.
