-
SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
Authors:
Jonathan Kutasov,
Yuqi Sun,
Paul Colognese,
Teun van der Weij,
Linda Petrini,
Chen Bo Calvin Zhang,
John Hughes,
Xiang Deng,
Henry Sleight,
Tyler Tracy,
Buck Shlegeris,
Joe Benton
Abstract:
As Large Language Models (LLMs) are increasingly deployed as autonomous agents in complex and long horizon settings, it is critical to evaluate their ability to sabotage users by pursuing hidden objectives. We study the ability of frontier LLMs to evade monitoring and achieve harmful hidden goals while completing a wide array of realistic tasks. We evaluate a broad range of frontier LLMs using SHA…
▽ More
As Large Language Models (LLMs) are increasingly deployed as autonomous agents in complex and long horizon settings, it is critical to evaluate their ability to sabotage users by pursuing hidden objectives. We study the ability of frontier LLMs to evade monitoring and achieve harmful hidden goals while completing a wide array of realistic tasks. We evaluate a broad range of frontier LLMs using SHADE (Subtle Harmful Agent Detection & Evaluation)-Arena, the first highly diverse agent evaluation dataset for sabotage and monitoring capabilities of LLM agents. SHADE-Arena consists of complex pairs of benign main tasks and harmful side objectives in complicated environments. Agents are evaluated on their ability to complete the side task without appearing suspicious to an LLM monitor. When measuring agent ability to (a) complete the main task, (b) complete the side task, and (c) avoid detection, we find that the best performing frontier models score 27% (Claude 3.7 Sonnet) and 15% (Gemini 2.5 Pro) as sabotage agents when overseen by Claude 3.6 Sonnet. For current frontier models, success on the side task relies heavily on having access to a hidden scratchpad that is not visible to the monitor. We also use SHADE-Arena to measure models' monitoring abilities, with the top monitor (Gemini 2.5 Pro) achieving an AUC of 0.87 at distinguishing benign and malign transcripts. We find that for now, models still struggle at sabotage due to failures in long-context main task execution. However, our measurements already demonstrate the difficulty of monitoring for subtle sabotage attempts, which we expect to only increase in the face of more complex and longer-horizon tasks.
△ Less
Submitted 17 June, 2025;
originally announced June 2025.
-
Minimizing entropy for translation surfaces
Authors:
Paul Colognese,
Mark Pollicott
Abstract:
In this note, we consider the entropy of unit area translation surfaces in the $SL(2, \mathbb R)$ orbits of square tiled surfaces that are the union of squares, where the singularities occur at the vertices and the singularities have a common cone angle. We show that the entropy over such orbits is minimized at those surfaces tiled by equilateral triangles where the singularities occur precisely a…
▽ More
In this note, we consider the entropy of unit area translation surfaces in the $SL(2, \mathbb R)$ orbits of square tiled surfaces that are the union of squares, where the singularities occur at the vertices and the singularities have a common cone angle. We show that the entropy over such orbits is minimized at those surfaces tiled by equilateral triangles where the singularities occur precisely at the vertices.
△ Less
Submitted 1 July, 2022; v1 submitted 15 October, 2021;
originally announced October 2021.
-
Volume growth for infinite graphs and translation surfaces
Authors:
Paul Colognese,
Mark Pollicott
Abstract:
In this note we give asymptotic estimates for the volume growth associated to suitable infinite graphs. Our main application is to give an asymptotic estimate for volume growth associated to translation surfaces.
In this note we give asymptotic estimates for the volume growth associated to suitable infinite graphs. Our main application is to give an asymptotic estimate for volume growth associated to translation surfaces.
△ Less
Submitted 30 July, 2021; v1 submitted 29 July, 2021;
originally announced July 2021.
-
The growth and distribution of large circles on translation surfaces
Authors:
Paul Colognese,
Mark Pollicott
Abstract:
We consider circles on a translation surface $X$, consisting of points joined to a common center point by a geodesic of length $R$. We show that as $R \to \infty$ these circles distribute to a measure on $X$ which is equivalent to the area. In the last section we consider analogous results for closed geodesics.
We consider circles on a translation surface $X$, consisting of points joined to a common center point by a geodesic of length $R$. We show that as $R \to \infty$ these circles distribute to a measure on $X$ which is equivalent to the area. In the last section we consider analogous results for closed geodesics.
△ Less
Submitted 29 July, 2021;
originally announced July 2021.
