COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act
Authors:
Philipp Guldimann,
Alexander Spiridonov,
Robin Staab,
Nikola Jovanović,
Mark Vero,
Velko Vechev,
Anna-Maria Gueorguieva,
Mislav Balunović,
Nikola Konstantinov,
Pavol Bielik,
Petar Tsankov,
Martin Vechev
Abstract:
The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requir…
▽ More
The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.
△ Less
Submitted 3 February, 2025; v1 submitted 10 October, 2024;
originally announced October 2024.
Enabling Input on Tiny/Headless Systems Using Morse Code
Authors:
Anna-Maria Gueorguieva,
Gulnar Rakhmetulla,
Ahmed Sabbir Arif
Abstract:
This paper presents results of a pilot study that explored the potential of Morse code as a method for text entry on mobile devices. In the study, participants without prior experience with Morse code reached 6.7 wpm with a Morse code keyboard in three short sessions. Learning was observed both in terms of text entry speed and accuracy, which suggests that the overall performance of the keyboard i…
▽ More
This paper presents results of a pilot study that explored the potential of Morse code as a method for text entry on mobile devices. In the study, participants without prior experience with Morse code reached 6.7 wpm with a Morse code keyboard in three short sessions. Learning was observed both in terms of text entry speed and accuracy, which suggests that the overall performance of the keyboard is likely to improve with practice.
△ Less
Submitted 11 December, 2020;
originally announced December 2020.
