Test & Evaluation Best Practices for Machine Learning-Enabled Systems
Authors:
Jaganmohan Chandrasekaran,
Tyler Cody,
Nicola McCarthy,
Erin Lanus,
Laura Freeman
Abstract:
Machine learning (ML) - based software systems are rapidly gaining adoption across various domains, making it increasingly essential to ensure they perform as intended. This report presents best practices for the Test and Evaluation (T&E) of ML-enabled software systems across its lifecycle. We categorize the lifecycle of ML-enabled software systems into three stages: component, integration and dep…
▽ More
Machine learning (ML) - based software systems are rapidly gaining adoption across various domains, making it increasingly essential to ensure they perform as intended. This report presents best practices for the Test and Evaluation (T&E) of ML-enabled software systems across its lifecycle. We categorize the lifecycle of ML-enabled software systems into three stages: component, integration and deployment, and post-deployment. At the component level, the primary objective is to test and evaluate the ML model as a standalone component. Next, in the integration and deployment stage, the goal is to evaluate an integrated ML-enabled system consisting of both ML and non-ML components. Finally, once the ML-enabled software system is deployed and operationalized, the T&E objective is to ensure the system performs as intended. Maintenance activities for ML-enabled software systems span the lifecycle and involve maintaining various assets of ML-enabled software systems.
Given its unique characteristics, the T&E of ML-enabled software systems is challenging. While significant research has been reported on T&E at the component level, limited work is reported on T&E in the remaining two stages. Furthermore, in many cases, there is a lack of systematic T&E strategies throughout the ML-enabled system's lifecycle. This leads practitioners to resort to ad-hoc T&E practices, which can undermine user confidence in the reliability of ML-enabled software systems. New systematic testing approaches, adequacy measurements, and metrics are required to address the T&E challenges across all stages of the ML-enabled system lifecycle.
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
A Survey of Data Security: Practices from Cybersecurity and Challenges of Machine Learning
Authors:
Padmaksha Roy,
Jaganmohan Chandrasekaran,
Erin Lanus,
Laura Freeman,
Jeremy Werner
Abstract:
Machine learning (ML) is increasingly being deployed in critical systems. The data dependence of ML makes securing data used to train and test ML-enabled systems of utmost importance. While the field of cybersecurity has well-established practices for securing information, ML-enabled systems create new attack vectors. Furthermore, data science and cybersecurity domains adhere to their own set of s…
▽ More
Machine learning (ML) is increasingly being deployed in critical systems. The data dependence of ML makes securing data used to train and test ML-enabled systems of utmost importance. While the field of cybersecurity has well-established practices for securing information, ML-enabled systems create new attack vectors. Furthermore, data science and cybersecurity domains adhere to their own set of skills and terminologies. This survey aims to present background information for experts in both domains in topics such as cryptography, access control, zero trust architectures, homomorphic encryption, differential privacy for machine learning, and federated learning to establish shared foundations and promote advancements in data security.
△ Less
Submitted 4 December, 2023; v1 submitted 6 October, 2023;
originally announced October 2023.
