-
Beyond Accuracy, SHAP, and Anchors -- On the difficulty of designing effective end-user explanations
Authors:
Zahra Abba Omar,
Nadia Nahar,
Jacob Tjaden,
Inès M. Gilles,
Fikir Mekonnen,
Jane Hsieh,
Christian Kästner,
Alka Menon
Abstract:
Modern machine learning produces models that are impossible for users or developers to fully understand -- raising concerns about trust, oversight and human dignity. Transparency and explainability methods aim to provide some help in understanding models, but it remains challenging for developers to design explanations that are understandable to target users and effective for their purpose. Emergi…
▽ More
Modern machine learning produces models that are impossible for users or developers to fully understand -- raising concerns about trust, oversight and human dignity. Transparency and explainability methods aim to provide some help in understanding models, but it remains challenging for developers to design explanations that are understandable to target users and effective for their purpose. Emerging guidelines and regulations set goals but may not provide effective actionable guidance to developers. In a controlled experiment with 124 participants, we investigate whether and how specific forms of policy guidance help developers design explanations for an ML-powered screening tool for diabetic retinopathy. Contrary to our expectations, we found that participants across the board struggled to produce quality explanations, comply with the provided policy requirements for explainability, and provide evidence of compliance. We posit that participant noncompliance is in part due to a failure to imagine and anticipate the needs of their audience, particularly non-technical stakeholders. Drawing on cognitive process theory and the sociological imagination to contextualize participants' failure, we recommend educational interventions.
△ Less
Submitted 28 January, 2025;
originally announced March 2025.
-
Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products
Authors:
Nadia Nahar,
Christian Kästner,
Jenna Butler,
Chris Parnin,
Thomas Zimmermann,
Christian Bird
Abstract:
Large Language Models (LLMs) are increasingly embedded into software products across diverse industries, enhancing user experiences, but at the same time introducing numerous challenges for developers. Unique characteristics of LLMs force developers, who are accustomed to traditional software development and evaluation, out of their comfort zones as the LLM components shatter standard assumptions…
▽ More
Large Language Models (LLMs) are increasingly embedded into software products across diverse industries, enhancing user experiences, but at the same time introducing numerous challenges for developers. Unique characteristics of LLMs force developers, who are accustomed to traditional software development and evaluation, out of their comfort zones as the LLM components shatter standard assumptions about software systems. This study explores the emerging solutions that software developers are adopting to navigate the encountered challenges. Leveraging a mixed-method research, including 26 interviews and a survey with 332 responses, the study identifies 19 emerging solutions regarding quality assurance that practitioners across several product teams at Microsoft are exploring. The findings provide valuable insights that can guide the development and evaluation of LLM-based products more broadly in the face of these challenges.
△ Less
Submitted 4 December, 2024; v1 submitted 15 October, 2024;
originally announced October 2024.
-
Automatic Speech Recognition for Biomedical Data in Bengali Language
Authors:
Shariar Kabir,
Nazmun Nahar,
Shyamasree Saha,
Mamunur Rashid
Abstract:
This paper presents the development of a prototype Automatic Speech Recognition (ASR) system specifically designed for Bengali biomedical data. Recent advancements in Bengali ASR are encouraging, but a lack of domain-specific data limits the creation of practical healthcare ASR models. This project bridges this gap by developing an ASR system tailored for Bengali medical terms like symptoms, sever…
▽ More
This paper presents the development of a prototype Automatic Speech Recognition (ASR) system specifically designed for Bengali biomedical data. Recent advancements in Bengali ASR are encouraging, but a lack of domain-specific data limits the creation of practical healthcare ASR models. This project bridges this gap by developing an ASR system tailored for Bengali medical terms like symptoms, severity levels, and diseases, encompassing two major dialects: Bengali and Sylheti. We train and evaluate two popular ASR frameworks on a comprehensive 46-hour Bengali medical corpus. Our core objective is to create deployable health-domain ASR systems for digital health applications, ultimately increasing accessibility for non-technical users in the healthcare sector.
△ Less
Submitted 16 June, 2024;
originally announced June 2024.
-
The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products
Authors:
Nadia Nahar,
Haoran Zhang,
Grace Lewis,
Shurui Zhou,
Christian Kästner
Abstract:
Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, hindering research progress to address these challenges. In this study, first and foremost, we contribute a dataset of 262 open-source ML products f…
▽ More
Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, hindering research progress to address these challenges. In this study, first and foremost, we contribute a dataset of 262 open-source ML products for end users (not just models), identified among more than half a million ML-related projects on GitHub. Then, we qualitatively and quantitatively analyze 30 open-source ML products to answer six broad research questions about development practices and system architecture. We find that the majority of the ML products in our sample represent more startup-style development than reported in past interview studies. We report 21 findings, including limited involvement of data scientists in many open-source ML products, unusually low modularity between ML and non-ML code, diverse architectural choices on incorporating models into products, and limited prevalence of industry best practices such as model testing, pipeline automation, and monitoring. Additionally, we discuss seven implications of this study on research, development, and education, including the need for tools to assist teams without data scientists, education opportunities, and open-source-specific research for privacy-preserving telemetry.
△ Less
Submitted 15 August, 2024; v1 submitted 8 August, 2023;
originally announced August 2023.
-
A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners
Authors:
Nadia Nahar,
Haoran Zhang,
Grace Lewis,
Shurui Zhou,
Christian Kästner
Abstract:
Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing challenges. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and presen…
▽ More
Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing challenges. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and present their collective findings, we conduct a meta-summary study: We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers. We highlight the most commonly reported challenges and hope this meta-summary will be a useful resource for the research community to prioritize research and education in this field.
△ Less
Submitted 31 March, 2023;
originally announced April 2023.
-
Aspirations and Practice of Model Documentation: Moving the Needle with Nudging and Traceability
Authors:
Avinash Bhat,
Austin Coursey,
Grace Hu,
Sixian Li,
Nadia Nahar,
Shurui Zhou,
Christian Kästner,
Jin L. C. Guo
Abstract:
The documentation practice for machine-learned (ML) models often falls short of established practices for traditional software, which impedes model accountability and inadvertently abets inappropriate or misuse of models. Recently, model cards, a proposal for model documentation, have attracted notable attention, but their impact on the actual practice is unclear. In this work, we systematically s…
▽ More
The documentation practice for machine-learned (ML) models often falls short of established practices for traditional software, which impedes model accountability and inadvertently abets inappropriate or misuse of models. Recently, model cards, a proposal for model documentation, have attracted notable attention, but their impact on the actual practice is unclear. In this work, we systematically study the model documentation in the field and investigate how to encourage more responsible and accountable documentation practice. Our analysis of publicly available model cards reveals a substantial gap between the proposal and the practice. We then design a tool named DocML aiming to (1) nudge the data scientists to comply with the model cards proposal during the model development, especially the sections related to ethics, and (2) assess and manage the documentation quality. A lab study reveals the benefit of our tool towards long-term documentation quality and accountability.
△ Less
Submitted 8 February, 2023; v1 submitted 13 April, 2022;
originally announced April 2022.
-
Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process
Authors:
Nadia Nahar,
Shurui Zhou,
Grace Lewis,
Christian Kästner
Abstract:
The introduction of machine learning (ML) components in software projects has created the need for software engineers to collaborate with data scientists and other specialists. While collaboration can always be challenging, ML introduces additional challenges with its exploratory model development process, additional skills and knowledge needed, difficulties testing ML systems, need for continuous…
▽ More
The introduction of machine learning (ML) components in software projects has created the need for software engineers to collaborate with data scientists and other specialists. While collaboration can always be challenging, ML introduces additional challenges with its exploratory model development process, additional skills and knowledge needed, difficulties testing ML systems, need for continuous evolution and monitoring, and non-traditional quality requirements such as fairness and explainability. Through interviews with 45 practitioners from 28 organizations, we identified key collaboration challenges that teams face when building and deploying ML systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges. We find that most of these challenges center around communication, documentation, engineering, and process and collect recommendations to address these challenges.
△ Less
Submitted 10 February, 2022; v1 submitted 19 October, 2021;
originally announced October 2021.
-
SysMap: A Lightweight Software Visualization Tool to Analyze the Software Evolution of a System
Authors:
Fazle Rabbi,
Nishat Tasnim Niloy,
Nadia Nahar,
Md. Nurul Ahad Tawhid
Abstract:
Software visualization helps to comprehend the system by providing a vivid illustration. The developers, as well as the analysts, can have a glance over the total system to understand the basic changes over time from a high-level point of view through this technique. In recent years, many tools are proposed to visualize software based on different architectural metaphors, such as as- solar system,…
▽ More
Software visualization helps to comprehend the system by providing a vivid illustration. The developers, as well as the analysts, can have a glance over the total system to understand the basic changes over time from a high-level point of view through this technique. In recent years, many tools are proposed to visualize software based on different architectural metaphors, such as as- solar system, city or park. Some of the solutions have just worked on system visualization where a few tried to explain the changes in software throughout different versions that still need heavy manual work. Keeping such limitations in mind, this paper proposes a lightweight tool named SysMap that takes the source codes of different versions of software systems, provides 3D illustrations of those systems and a graphical statistic of its evolution. To build the graphical element to represent the system, the source code has been studied to find out different software metrics. For experimentation, several open-source java projects were chosen to find out the necessary information. Henceforth, this tool will surely increase the work efficiency of both the developer and analyst by reducing the manual effort and by providing the graphical view to comprehend the software evolution over time.
△ Less
Submitted 12 August, 2021;
originally announced August 2021.
-
The Influences of Pre-birth Factors in Early Assessment of Child Mortality using Machine Learning Techniques
Authors:
Asadullah Hill Galib,
Nadia Nahar,
B M Mainul Hossain
Abstract:
Analysis of child mortality is crucial as it pertains to the policy and programs of a country. The early assessment of patterns and trends in causes of child mortality help decision-makers assess needs, prioritize interventions, and monitor progress. Post-birth factors of the child, such as real-time clinical data, health data of the child, etc. are frequently used in child mortality studies. Howe…
▽ More
Analysis of child mortality is crucial as it pertains to the policy and programs of a country. The early assessment of patterns and trends in causes of child mortality help decision-makers assess needs, prioritize interventions, and monitor progress. Post-birth factors of the child, such as real-time clinical data, health data of the child, etc. are frequently used in child mortality studies. However, in the early assessment of child mortality, pre-birth factors would be more practical and beneficial than the post-birth factors. This study aims at incorporating pre-birth factors, such as birth history, maternal history, reproduction history, socioeconomic condition, etc. for classifying child mortality. To assess the relative importance of the features, Information Gain (IG) attribute evaluator is employed. For classifying child mortality, four machine learning algorithms are evaluated. Results show that the proposed approach achieved an AUC score of 0.947 in classifying child mortality which outperformed the clinical standards. In terms of accuracy, precision, recall, and f-1 score, the results are also notable and uniform. In developing countries like Bangladesh, the early assessment of child mortality using pre-birth factors would be effective and feasible as it avoids the uncertainty of the post-birth factors.
△ Less
Submitted 18 November, 2020;
originally announced November 2020.
-
Watchword-Oriented and Time-Stamped Algorithms for Tamper-Proof Cloud Provenance Cognition
Authors:
Asif Imran,
Nadia Nahar,
Kazi Sakib
Abstract:
Provenance is derivative journal information about the origin and activities of system data and processes. For a highly dynamic system like the cloud, provenance can be accurately detected and securely used in cloud digital forensic investigation activities. This paper proposes watchword oriented provenance cognition algorithm for the cloud environment. Additionally time-stamp based buffer verifyi…
▽ More
Provenance is derivative journal information about the origin and activities of system data and processes. For a highly dynamic system like the cloud, provenance can be accurately detected and securely used in cloud digital forensic investigation activities. This paper proposes watchword oriented provenance cognition algorithm for the cloud environment. Additionally time-stamp based buffer verifying algorithm is proposed for securing the access to the detected cloud provenance. Performance analysis of the novel algorithms proposed here yields a desirable detection rate of 89.33% and miss rate of 8.66%. The securing algorithm successfully rejects 64% of malicious requests, yielding a cumulative frequency of 21.43 for MR.
△ Less
Submitted 19 September, 2014;
originally announced September 2014.
