-
The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System
Authors:
Anton Hummel,
Håkan Burden,
Susanne Stenberg,
Jan-Philipp Steghöfer,
Niklas Kühl
Abstract:
Explainable AI (XAI) is a promising solution to ensure compliance with the EU AI Act, the first multi-national regulation for AI. XAI aims to enhance transparency and human oversight of AI systems, particularly ``black-box models'', which are criticized as incomprehensible. However, the discourse around the main stakeholders in the AI Act and XAI appears disconnected. While XAI prioritizes the end…
▽ More
Explainable AI (XAI) is a promising solution to ensure compliance with the EU AI Act, the first multi-national regulation for AI. XAI aims to enhance transparency and human oversight of AI systems, particularly ``black-box models'', which are criticized as incomprehensible. However, the discourse around the main stakeholders in the AI Act and XAI appears disconnected. While XAI prioritizes the end user's needs as the primary goal, the AI Act focuses on the obligations of the provider and deployer of the AI system. We aim to bridge this divide and provide guidance on how these two worlds are related. By fostering an interdisciplinary discussion in a cross-functional team with XAI, AI Act, legal, and requirements engineering experts, we walk through the steps necessary to analyze an AI-based clinical decision support system to clarify the end-user needs and assess AI Act applicability. By analyzing our justified understanding using an AI system under development as a case, we show that XAI techniques can fill a gap between stakeholder needs and the requirements of the AI Act. We look at the similarities and contrasts between the legal requirements and the needs of stakeholders. In doing so, we encourage researchers and practitioners from the XAI community to reflect on their role towards the AI Act by achieving a mutual understanding of the implications of XAI and the AI Act within different disciplines.
△ Less
Submitted 22 May, 2025;
originally announced May 2025.
-
A sandbox study proposal for private and distributed health data analysis
Authors:
Rickard Brännvall,
Hanna Svensson,
Kannaki Kaliyaperumal,
Håkan Burden,
Susanne Stenberg
Abstract:
This paper presents a sandbox study proposal focused on the distributed processing of personal health data within the Vinnova-funded SARDIN project. The project aims to develop the Health Data Bank (Hälsodatabanken in Swedish), a secure platform for research and innovation that complies with the European Health Data Space (EHDS) legislation. By minimizing the sharing and storage of personal data,…
▽ More
This paper presents a sandbox study proposal focused on the distributed processing of personal health data within the Vinnova-funded SARDIN project. The project aims to develop the Health Data Bank (Hälsodatabanken in Swedish), a secure platform for research and innovation that complies with the European Health Data Space (EHDS) legislation. By minimizing the sharing and storage of personal data, the platform sends analysis tasks directly to the original data locations, avoiding centralization. This approach raises questions about data controller responsibilities in distributed environments and the anonymization status of aggregated statistical results. The study explores federated analysis, secure multi-party aggregation, and differential privacy techniques, informed by real-world examples from clinical research on Parkinson's disease, stroke rehabilitation, and wound analysis. To validate the proposed study, numerical experiments were conducted using four open-source datasets to assess the feasibility and effectiveness of the proposed methods. The results support the methods for the proposed sandbox study by demonstrating that differential privacy in combination with secure aggregation techniques significantly improves the privacy-utility trade-off.
△ Less
Submitted 24 January, 2025;
originally announced January 2025.
-
Involving External Stakeholders in Project Courses
Authors:
Jan-Philipp Steghöfer,
Håkan Burden,
Regina Hebig,
Gul Calikli,
Robert Feldt,
Imed Hammouda,
Jennifer Horkoff,
Eric Knauss,
Grischa Liebel
Abstract:
Problem: The involvement of external stakeholders in capstone projects and project courses is desirable due to its potential positive effects on the students. Capstone projects particularly profit from the inclusion of an industrial partner to make the project relevant and help students acquire professional skills. In addition, an increasing push towards education that is aligned with industry and…
▽ More
Problem: The involvement of external stakeholders in capstone projects and project courses is desirable due to its potential positive effects on the students. Capstone projects particularly profit from the inclusion of an industrial partner to make the project relevant and help students acquire professional skills. In addition, an increasing push towards education that is aligned with industry and incorporates industrial partners can be observed. However, the involvement of external stakeholders in teaching moments can create friction and could, in the worst case, lead to frustration of all involved parties. Contribution: We developed a model that allows analysing the involvement of external stakeholders in university courses both in a retrospective fashion, to gain insights from past course instances, and in a constructive fashion, to plan the involvement of external stakeholders. Key Concepts: The conceptual model and the accompanying guideline guide the teachers in their analysis of stakeholder involvement. The model is comprised of several activities (define, execute, and evaluate the collaboration). The guideline provides questions that the teachers should answer for each of these activities. In the constructive use, the model allows teachers to define an action plan based on an analysis of potential stakeholders and the pedagogical objectives. In the retrospective use, the model allows teachers to identify issues that appeared during the project and their underlying causes. Drawing from ideas of the reflective practitioner, the model contains an emphasis on reflection and interpretation of the observations made by the teacher and other groups involved in the courses. Key Lessons: Applying the model retrospectively to a total of eight courses shows that it is possible to reveal hitherto implicit risks and assumptions and to gain a better insight into the interaction...
△ Less
Submitted 4 May, 2018; v1 submitted 3 May, 2018;
originally announced May 2018.
