-
Machine Learning Security in Industry: A Quantitative Survey
Authors:
Kathrin Grosse,
Lukas Bieringer,
Tarek Richard Besold,
Battista Biggio,
Katharina Krombholz
Abstract:
Despite the large body of academic work on machine learning security, little is known about the occurrence of attacks on machine learning systems in the wild. In this paper, we report on a quantitative study with 139 industrial practitioners. We analyze attack occurrence and concern and evaluate statistical hypotheses on factors influencing threat perception and exposure. Our results shed light on…
▽ More
Despite the large body of academic work on machine learning security, little is known about the occurrence of attacks on machine learning systems in the wild. In this paper, we report on a quantitative study with 139 industrial practitioners. We analyze attack occurrence and concern and evaluate statistical hypotheses on factors influencing threat perception and exposure. Our results shed light on real-world attacks on deployed machine learning. On the organizational level, while we find no predictors for threat exposure in our sample, the amount of implement defenses depends on exposure to threats or expected likelihood to become a target. We also provide a detailed analysis of practitioners' replies on the relevance of individual machine learning attacks, unveiling complex concerns like unreliable decision making, business information leakage, and bias introduction into models. Finally, we find that on the individual level, prior knowledge about machine learning security influences threat perception. Our work paves the way for more research about adversarial machine learning in practice, but yields also insights for regulation and auditing.
△ Less
Submitted 10 March, 2023; v1 submitted 11 July, 2022;
originally announced July 2022.
-
Mental Models of Adversarial Machine Learning
Authors:
Lukas Bieringer,
Kathrin Grosse,
Michael Backes,
Battista Biggio,
Katharina Krombholz
Abstract:
Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root ca…
▽ More
Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root causes or improve risk communication. Our study reveals two \facets of practitioners' mental models of machine learning security. Firstly, practitioners often confuse machine learning security with threats and defences that are not directly related to machine learning. Secondly, in contrast to most academic research, our participants perceive security of machine learning as not solely related to individual models, but rather in the context of entire workflows that consist of multiple components. Jointly with our additional findings, these two facets provide a foundation to substantiate mental models for machine learning security and have implications for the integration of adversarial machine learning into corporate workflows, \new{decreasing practitioners' reported uncertainty}, and appropriate regulatory frameworks for machine learning security.
△ Less
Submitted 29 June, 2022; v1 submitted 8 May, 2021;
originally announced May 2021.
-
Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators
Authors:
Christian Tiefenau,
Maximilian Häring,
Katharina Krombholz,
Emanuel von Zezschwitz
Abstract:
Experts agree that keeping systems up to date is a powerful security measure. Previous work found that users sometimes explicitly refrain from performing timely updates, e.g., due to bad experiences which has a negative impact on end-user security. Another important user group has been investigated less extensively: system administrators, who are responsible for keeping complex and heterogeneous s…
▽ More
Experts agree that keeping systems up to date is a powerful security measure. Previous work found that users sometimes explicitly refrain from performing timely updates, e.g., due to bad experiences which has a negative impact on end-user security. Another important user group has been investigated less extensively: system administrators, who are responsible for keeping complex and heterogeneous system landscapes available and secure.
In this paper, we sought to understand administrators' behavior, experiences, and attitudes regarding updates in a corporate environment. Based on the results of an interview study, we developed an online survey and quantified common practices and obstacles (e.g., downtime or lack of information about updates). The findings indicate that even experienced administrators struggle with update processes as the consequences of an update are sometimes hard to assess. Therefore, we argue that more usable monitoring and update processes are essential to guarantee IT security at scale.
△ Less
Submitted 17 July, 2020;
originally announced July 2020.
-
VisualPhishNet: Zero-Day Phishing Website Detection by Visual Similarity
Authors:
Sahar Abdelnabi,
Katharina Krombholz,
Mario Fritz
Abstract:
Phishing websites are still a major threat in today's Internet ecosystem. Despite numerous previous efforts, similarity-based detection methods do not offer sufficient protection for the trusted websites - in particular against unseen phishing pages. This paper contributes VisualPhishNet, a new similarity-based phishing detection framework, based on a triplet Convolutional Neural Network (CNN). Vi…
▽ More
Phishing websites are still a major threat in today's Internet ecosystem. Despite numerous previous efforts, similarity-based detection methods do not offer sufficient protection for the trusted websites - in particular against unseen phishing pages. This paper contributes VisualPhishNet, a new similarity-based phishing detection framework, based on a triplet Convolutional Neural Network (CNN). VisualPhishNet learns profiles for websites in order to detect phishing websites by a similarity metric that can generalize to pages with new visual appearances. We furthermore present VisualPhish, the largest dataset to date that facilitates visual phishing detection in an ecologically valid manner. We show that our method outperforms previous visual similarity phishing detection approaches by a large margin while being robust against a range of evasion attacks.
△ Less
Submitted 5 July, 2020; v1 submitted 31 August, 2019;
originally announced September 2019.
-
Body Shape Privacy in Images: Understanding Privacy and Preventing Automatic Shape Extraction
Authors:
Hosnieh Sattar,
Katharina Krombholz,
Gerard Pons-Moll,
Mario Fritz
Abstract:
Modern approaches to pose and body shape estimation have recently achieved strong performance even under challenging real-world conditions. Even from a single image of a clothed person, a realistic looking body shape can be inferred that captures a users' weight group and body shape type well. This opens up a whole spectrum of applications -- in particular in fashion -- where virtual try-on and re…
▽ More
Modern approaches to pose and body shape estimation have recently achieved strong performance even under challenging real-world conditions. Even from a single image of a clothed person, a realistic looking body shape can be inferred that captures a users' weight group and body shape type well. This opens up a whole spectrum of applications -- in particular in fashion -- where virtual try-on and recommendation systems can make use of these new and automatized cues. However, a realistic depiction of the undressed body is regarded highly private and therefore might not be consented by most people. Hence, we ask if the automatic extraction of such information can be effectively evaded. While adversarial perturbations have been shown to be effective for manipulating the output of machine learning models -- in particular, end-to-end deep learning approaches -- state of the art shape estimation methods are composed of multiple stages. We perform the first investigation of different strategies that can be used to effectively manipulate the automatic shape estimation while preserving the overall appearance of the original image.
△ Less
Submitted 22 October, 2020; v1 submitted 27 May, 2019;
originally announced May 2019.
