-
LLMSecCode: Evaluating Large Language Models for Secure Coding
Authors:
Anton Rydén,
Erik Näslund,
Elad Michael Schiller,
Magnus Almgren
Abstract:
The rapid deployment of Large Language Models (LLMs) requires careful consideration of their effect on cybersecurity. Our work aims to improve the selection process of LLMs that are suitable for facilitating Secure Coding (SC). This raises challenging research questions, such as (RQ1) Which functionality can streamline the LLM evaluation? (RQ2) What should the evaluation measure? (RQ3) How to atte…
▽ More
The rapid deployment of Large Language Models (LLMs) requires careful consideration of their effect on cybersecurity. Our work aims to improve the selection process of LLMs that are suitable for facilitating Secure Coding (SC). This raises challenging research questions, such as (RQ1) Which functionality can streamline the LLM evaluation? (RQ2) What should the evaluation measure? (RQ3) How to attest that the evaluation process is impartial? To address these questions, we introduce LLMSecCode, an open-source evaluation framework designed to assess LLM SC capabilities objectively.
We validate the LLMSecCode implementation through experiments. When varying parameters and prompts, we find a 10% and 9% difference in performance, respectively. We also compare some results to reliable external actors, where our results show a 5% difference.
We strive to ensure the ease of use of our open-source framework and encourage further development by external actors. With LLMSecCode, we hope to encourage the standardization and benchmarking of LLMs' capabilities in security-oriented code and tasks.
△ Less
Submitted 28 August, 2024;
originally announced August 2024.
-
Security-Enhancing Digital Twins: Characteristics, Indicators, and Future Perspectives
Authors:
Matthias Eckhart,
Andreas Ekelhart,
David Allison,
Magnus Almgren,
Katharina Ceesay-Seitz,
Helge Janicke,
Simin Nadjm-Tehrani,
Awais Rashid,
Mark Yampolskiy
Abstract:
The term "digital twin" (DT) has become a key theme of the cyber-physical systems (CPSs) area, while remaining vaguely defined as a virtual replica of an entity. This article identifies DT characteristics essential for enhancing CPS security and discusses indicators to evaluate them.
The term "digital twin" (DT) has become a key theme of the cyber-physical systems (CPSs) area, while remaining vaguely defined as a virtual replica of an entity. This article identifies DT characteristics essential for enhancing CPS security and discusses indicators to evaluate them.
△ Less
Submitted 2 June, 2023; v1 submitted 30 April, 2023;
originally announced May 2023.
-
Evaluation of Open-source Tools for Differential Privacy
Authors:
Shiliang Zhang,
Anton Hagermalm,
Sanjin Slavnic,
Elad Michael Schiller,
Magnus Almgren
Abstract:
Differential privacy (DP) defines privacy protection by promising quantified indistinguishability between individuals that consent to share their privacy-sensitive information and the ones that do not. DP aims to deliver this promise by including well-crafted elements of random noise in the published data and thus there is an inherent trade-off between the degree of privacy protection and the abil…
▽ More
Differential privacy (DP) defines privacy protection by promising quantified indistinguishability between individuals that consent to share their privacy-sensitive information and the ones that do not. DP aims to deliver this promise by including well-crafted elements of random noise in the published data and thus there is an inherent trade-off between the degree of privacy protection and the ability to utilize the protected data. Currently, several open-source tools were proposed for DP provision. To the best of our knowledge, there is no comprehensive study for comparing these open-source tools with respect to their ability to balance DP's inherent trade-off as well as the use of system resources. This work proposes an open-source evaluation framework for privacy protection solutions and offers evaluation for OpenDP Smartnoise, Google DP, PyTorch Opacus, Tensorflow Privacy, and Diffprivlib. In addition to studying their ability to balance the above trade-off, we consider discrete and continuous attributes by quantifying their performance under different data sizes. Our results reveal several patterns that developers should have in mind when selecting tools under different application needs and criteria. This evaluation survey can be the basis for an improved selection of open-source DP tools and quicker adaptation of DP.
△ Less
Submitted 24 May, 2022; v1 submitted 19 February, 2022;
originally announced February 2022.
-
CASAD: CAN-Aware Stealthy-Attack Detection for In-Vehicle Networks
Authors:
Nasser Nowdehi,
Wissam Aoudi,
Magnus Almgren,
Tomas Olovsson
Abstract:
Nowadays, vehicles have complex in-vehicle networks (IVNs) with millions of lines of code controlling almost every function in the vehicle including safety-critical functions. It has recently been shown that IVNs are becoming increasingly vulnerable to cyber-attacks capable of taking control of vehicles, thereby threatening the safety of the passengers. Several countermeasures have been proposed i…
▽ More
Nowadays, vehicles have complex in-vehicle networks (IVNs) with millions of lines of code controlling almost every function in the vehicle including safety-critical functions. It has recently been shown that IVNs are becoming increasingly vulnerable to cyber-attacks capable of taking control of vehicles, thereby threatening the safety of the passengers. Several countermeasures have been proposed in the literature in response to the arising threats, however, hurdle requirements imposed by the industry is hindering their adoption in practice. In particular, detecting attacks on IVNs is challenged by strict resource constraints and utterly complex communication patterns that vary even for vehicles of the same model. In addition, existing solutions suffer from two main drawbacks. First, they depend on the underlying vehicle configuration, and second, they are incapable of detecting certain attacks of a stealthy nature. In this paper, we propose CASAD, a CAN-Aware Stealthy-Attack Detection mechanism that does not abide by the strict specifications predefined for every vehicle model and addresses key real-world deployability challenges. Our fast, lightweight, and system-agnostic approach learns the normal behavior of IVN dynamics from historical data and detects deviations by continuously monitoring IVN traffic. We demonstrate the effectiveness of CASAD by conducting various experiments on a CAN bus prototype, a 2018 Volvo XC60, and publicly available data from two real vehicles. Our approach is experimentally shown to be effective against different attack scenarios, including the prompt detection of stealthy attacks, and has considerable potential applicability to real vehicles.
△ Less
Submitted 18 September, 2019;
originally announced September 2019.
