Showing 1–1 of 1 results for author: Panchal, H S

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

Authors: Qiusi Zhan, Richard Fang, Henil Shalin Panchal, Daniel Kang

Abstract: Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In th… ▽ More Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent. △ Less

Submitted 3 March, 2025; v1 submitted 26 February, 2025; originally announced March 2025.

Comments: 17 pages, 5 figures, 6 tables (NAACL 2025 Findings)