ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
Authors:
Saurabh Jha,
Rohan Arora,
Yuji Watanabe,
Takumi Yanagawa,
Yinfang Chen,
Jackson Clark,
Bhavya Bhavya,
Mudit Verma,
Harshit Kumar,
Hirokuni Kitahara,
Noah Zheutlin,
Saki Takano,
Divya Pathak,
Felix George,
Xinbo Wu,
Bekir O. Turkkan,
Gerard Vanloo,
Michael Nidd,
Ting Dai,
Oishik Chatterjee,
Pranjal Gupta,
Suranjana Samanta,
Pooja Aggarwal,
Rong Lee,
Pavankumar Murali
, et al. (18 additional authors not shown)
Abstract:
Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Securit…
▽ More
Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.
△ Less
Submitted 7 February, 2025;
originally announced February 2025.
Optimal Workload Placement on Multi-Instance GPUs
Authors:
Bekir Turkkan,
Pavankumar Murali,
Pavithra Harsha,
Rohan Arora,
Gerard Vanloo,
Chandra Narayanaswami
Abstract:
There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we in…
▽ More
There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we investigate how to optimize the placement of Large Language Model (LLM)-based AI Inferencing workloads on GPUs. We first identify and present several use cases that are encountered in practice that require workloads to be efficiently placed or migrated to other GPUs to make room for incoming workloads. The overarching goal is to use as few GPUs as possible and to further minimize memory and compute wastage on GPUs that are utilized. We have developed two approaches to address this problem: an optimization method and a heuristic method. We benchmark these with two workload scheduling heuristics for multiple use cases. Our results show up to 2.85x improvement in the number of GPUs used and up to 70% reduction in GPU wastage over baseline heuristics. We plan to enable the SRE community to leverage our proposed method in production environments.
△ Less
Submitted 10 September, 2024;
originally announced September 2024.
