Showing 1–8 of 8 results for author: Kortuem, G

Understanding Mental Models of Generative Conversational Search and The Effect of Interface Transparency

Authors: Chadha Degachi, Samuel Kernan Freire, Evangelos Niforatos, Gerd Kortuem

Abstract: The experience and adoption of conversational search is tied to the accuracy and completeness of users' mental models -- their internal frameworks for understanding and predicting system behaviour. Thus, understanding these models can reveal areas for design interventions. Transparency is one such intervention which can improve system interpretability and enable mental model alignment. While past… ▽ More The experience and adoption of conversational search is tied to the accuracy and completeness of users' mental models -- their internal frameworks for understanding and predicting system behaviour. Thus, understanding these models can reveal areas for design interventions. Transparency is one such intervention which can improve system interpretability and enable mental model alignment. While past research has explored mental models of search engines, those of generative conversational search remain underexplored, even while the popularity of these systems soars. To address this, we conducted a study with 16 participants, who performed 4 search tasks using 4 conversational interfaces of varying transparency levels. Our analysis revealed that most user mental models were too abstract to support users in explaining individual search instances. These results suggest that 1) mental models may pose a barrier to appropriate trust in conversational search, and 2) hybrid web-conversational search is a promising novel direction for future search interface design. △ Less

Submitted 4 June, 2025; originally announced June 2025.

Comments: Work in Progress

"A Great Start, But...": Evaluating LLM-Generated Mind Maps for Information Mapping in Video-Based Design

Authors: Tianhao He, Karthi Saravanan, Evangelos Niforatos, Gerd Kortuem

Abstract: Extracting concepts and understanding relationships from videos is essential in Video-Based Design (VBD), where videos serve as a primary medium for exploration but require significant effort in managing meta-information. Mind maps, with their ability to visually organize complex data, offer a promising approach for structuring and analysing video content. Recent advancements in Large Language Mod… ▽ More Extracting concepts and understanding relationships from videos is essential in Video-Based Design (VBD), where videos serve as a primary medium for exploration but require significant effort in managing meta-information. Mind maps, with their ability to visually organize complex data, offer a promising approach for structuring and analysing video content. Recent advancements in Large Language Models (LLMs) provide new opportunities for meta-information processing and visual understanding in VBD, yet their application remains underexplored. This study recruited 28 VBD practitioners to investigate the use of prompt-tuned LLMs for generating mind maps from ethnographic videos. Comparing LLM-generated mind maps with those created by professional designers, we evaluated rated scores, design effectiveness, and user experience across two contexts. Findings reveal that LLMs effectively capture central concepts but struggle with hierarchical organization and contextual grounding. We discuss trust, customization, and workflow integration as key factors to guide future research on LLM-supported information mapping in VBD. △ Less

Submitted 16 January, 2025; originally announced January 2025.

Tangi: a Tool to Create Tangible Artifacts for Sharing Insights from 360$^\circ$ Video

Authors: Wo Meijer, Jacky Bourgeois, Tilman Dingler, Gerd Kortuem

Abstract: Designers often engage with video to gain rich, temporal insights about the context of users, collaboratively analyzing it to gather ideas, challenge assumptions, and foster empathy. To capture the full visual context of users and their situations, designers are adopting 360$^\circ$ video, providing richer, more multi-layered insights. Unfortunately, the spherical nature of 360$^\circ$ video means… ▽ More Designers often engage with video to gain rich, temporal insights about the context of users, collaboratively analyzing it to gather ideas, challenge assumptions, and foster empathy. To capture the full visual context of users and their situations, designers are adopting 360$^\circ$ video, providing richer, more multi-layered insights. Unfortunately, the spherical nature of 360$^\circ$ video means designers cannot create tangible video artifacts such as storyboards for collaborative analysis. To overcome this limitation, we created Tangi, a web-based tool that converts 360$^\circ$ images into tangible 360$^\circ$ video artifacts, that enable designers to embody and share their insights. Our evaluation with nine experienced designers demonstrates that the artifacts Tangi creates enable tangible interactions found in collaborative workshops and introduce two new capabilities: spatial orientation within 360$^\circ$ environments and linking specific details to the broader 360$^\circ$ context. Since Tangi is an open-source tool, designers can immediately leverage 360$^\circ$ video in collaborative workshops. △ Less

Submitted 15 November, 2024; originally announced November 2024.

DesignMinds: Enhancing Video-Based Design Ideation with Vision-Language Model and Context-Injected Large Language Model

Authors: Tianhao He, Andrija Stankovic, Evangelos Niforatos, Gerd Kortuem

Abstract: Ideation is a critical component of video-based design (VBD), where videos serve as the primary medium for design exploration and inspiration. The emergence of generative AI offers considerable potential to enhance this process by streamlining video analysis and facilitating idea generation. In this paper, we present DesignMinds, a prototype that integrates a state-of-the-art Vision-Language Model… ▽ More Ideation is a critical component of video-based design (VBD), where videos serve as the primary medium for design exploration and inspiration. The emergence of generative AI offers considerable potential to enhance this process by streamlining video analysis and facilitating idea generation. In this paper, we present DesignMinds, a prototype that integrates a state-of-the-art Vision-Language Model (VLM) with a context-enhanced Large Language Model (LLM) to support ideation in VBD. To evaluate DesignMinds, we conducted a between-subject study with 35 design practitioners, comparing its performance to a baseline condition. Our results demonstrate that DesignMinds significantly enhances the flexibility and originality of ideation, while also increasing task engagement. Importantly, the introduction of this technology did not negatively impact user experience, technology acceptance, or usability. △ Less

Submitted 6 November, 2024; originally announced November 2024.

Sphere Window: Challenges and Opportunities of 360° Video in Collaborative Design Workshops

Authors: Wo Meijer, Jacky Bourgeois, Wilhelm Frederik van der Vegte, Gerd Kortuem

Abstract: The increased ubiquity of 360° video presents a unique opportunity for designers to deeply engage with the world of users by capturing the complete visual context. However, the opportunities and challenges 360° video introduces for video design ethnography is unclear. This study investigates this gap through 16 workshops in which experienced designers engaged with 360° video. Our analysis shows th… ▽ More The increased ubiquity of 360° video presents a unique opportunity for designers to deeply engage with the world of users by capturing the complete visual context. However, the opportunities and challenges 360° video introduces for video design ethnography is unclear. This study investigates this gap through 16 workshops in which experienced designers engaged with 360° video. Our analysis shows that while 360° video enhances designers' ability to explore and understand user contexts, it also complicates the process of sharing insights. To address this challenge, we present two opportunities to support the use of 360° video by designers - the creation of designerly 360° video annotation tools, and 360° ``screenshots'' - in order to enable designers to leverage the complete context of 360° video for user research. △ Less

Submitted 17 July, 2024; originally announced July 2024.

MarkupLens: An AI-Powered Tool to Support Designers in Video-Based Analysis at Scale

Authors: Tianhao He, Ying Zhang, Evangelos Niforatos, Gerd Kortuem

Abstract: Video-Based Design (VBD) is a design methodology that utilizes video as a primary tool for understanding user interactions, prototyping, and conducting research to enhance the design process. Artificial Intelligence (AI) can be instrumental in video-based design by analyzing and interpreting visual data from videos to enhance user interaction, automate design processes, and improve product functio… ▽ More Video-Based Design (VBD) is a design methodology that utilizes video as a primary tool for understanding user interactions, prototyping, and conducting research to enhance the design process. Artificial Intelligence (AI) can be instrumental in video-based design by analyzing and interpreting visual data from videos to enhance user interaction, automate design processes, and improve product functionality. In this study, we explore how AI can enhance professional video-based design with a State-of-the-Art (SOTA) deep learning model. We developed a prototype annotation platform (MarkupLens) and conducted a between-subjects eye-tracking study with 36 designers, annotating videos with three levels of AI assistance. Our findings indicate that MarkupLens improved design annotation quality and productivity. Additionally, it reduced the cognitive load that designers exhibited and enhanced their User Experience (UX). We believe that designer-AI collaboration can greatly enhance the process of eliciting insights in video-based design. △ Less

Submitted 8 March, 2024; originally announced March 2024.

Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute

Authors: Kars Alfrink, Ianus Keller, Neelke Doorn, Gerd Kortuem

Abstract: Local governments increasingly use artificial intelligence (AI) for automated decision-making. Contestability, making systems responsive to dispute, is a way to ensure they respect human rights to autonomy and dignity. We investigate the design of public urban AI systems for contestability through the example of camera cars: human-driven vehicles equipped with image sensors. Applying a provisional… ▽ More Local governments increasingly use artificial intelligence (AI) for automated decision-making. Contestability, making systems responsive to dispute, is a way to ensure they respect human rights to autonomy and dignity. We investigate the design of public urban AI systems for contestability through the example of camera cars: human-driven vehicles equipped with image sensors. Applying a provisional framework for contestable AI, we use speculative design to create a concept video of a contestable camera car. Using this concept video, we then conduct semi-structured interviews with 17 civil servants who work with AI employed by a large northwestern European city. The resulting data is analyzed using reflexive thematic analysis to identify the main challenges facing the implementation of contestability in public AI. We describe how civic participation faces issues of representation, public AI systems should integrate with existing democratic practices, and cities must expand capacities for responsible AI development and operation. △ Less

Submitted 9 February, 2023; originally announced February 2023.

Comments: Conditionally accepted to CHI 2023

Micro-Navigation for Urban Bus Passengers: Using the Internet of Things to Improve the Public Transport Experience

Authors: Stefan Foell, Gerd Kortuem, Reza Rawassizadeh, Marcus Handte, Umer Iqbal, Pedro Marron

Abstract: Public bus services are widely deployed in cities around the world because they provide cost-effective and economic public transportation. However, from a passenger point of view urban bus systems can be complex and difficult to navigate, especially for disadvantaged users, i.e. tourists, novice users, older people, and people with impaired cognitive or physical abilities. We present Urban Bus Nav… ▽ More Public bus services are widely deployed in cities around the world because they provide cost-effective and economic public transportation. However, from a passenger point of view urban bus systems can be complex and difficult to navigate, especially for disadvantaged users, i.e. tourists, novice users, older people, and people with impaired cognitive or physical abilities. We present Urban Bus Navigator (UBN), a reality-aware urban navigation system for bus passengers with the ability to recognize and track the physical public transport infrastructure such as buses. Unlike traditional location-aware mobile transport applications, UBN acts as a true navigation assistant for public transport users. Insights from a six-month long trial in Madrid indicate that UBN removes barriers for public transport usage and has a positive impact on how people feel about public transport journeys. △ Less

Submitted 15 February, 2015; v1 submitted 20 December, 2014; originally announced December 2014.

Comments: Urb-IoT 2014