-
Beyond the Individual: A Community-Engaged Framework for Ethical Online Community Research
Authors:
Matthew Zent,
Seraphina Yong,
Dhruv Bala,
Stevie Chancellor,
Joseph A. Konstan,
Loren Terveen,
Svetlana Yarosh
Abstract:
Online community research routinely poses minimal risk to individuals, but does the same hold true for online communities? In response to high-profile breaches of online community trust and increased debate in the social computing research community on the ethics of online community research, this paper investigates community-level harms and benefits of research. Through 9 participatory-inspired w…
▽ More
Online community research routinely poses minimal risk to individuals, but does the same hold true for online communities? In response to high-profile breaches of online community trust and increased debate in the social computing research community on the ethics of online community research, this paper investigates community-level harms and benefits of research. Through 9 participatory-inspired workshops with four critical online communities (Wikipedia, InTheRooms, CaringBridge, and r/AskHistorians) we found researchers should engage more directly with communities' primary purpose by rationalizing their methods and contributions in the context of community goals to equalize the beneficiaries of community research. To facilitate deeper alignment of these expectations, we present the FACTORS (Functions for Action with Communities: Teaching, Overseeing, Reciprocating, and Sustaining) framework for ethical online community research. Finally, we reflect on our findings by providing implications for researchers and online communities to identify and implement functions for navigating community-level harms and benefits.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Large Language Models as Conversational Movie Recommenders: A User Study
Authors:
Ruixuan Sun,
Xinyi Li,
Avinash Akella,
Joseph A. Konstan
Abstract:
This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active us…
▽ More
This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.
△ Less
Submitted 29 April, 2024;
originally announced April 2024.
-
What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders
Authors:
Ruixuan Sun,
Xinyi Wu,
Avinash Akella,
Ruoyan Kong,
Bart Knijnenburg,
Joseph A. Konstan
Abstract:
In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We…
▽ More
In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items and weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. To further explain the reasons behind the underperformance, we apply a comprehensive path analysis. We discover that the lack of diversity and too much serendipity from DL models can negatively impact the consequent perceived transparency and personalization of recommendations. Such a path ultimately leads to lower summative user satisfaction. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure a good user experience, while their demands for transparency and trust can not be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.
△ Less
Submitted 1 May, 2024; v1 submitted 21 January, 2024;
originally announced January 2024.
-
COVID-19 as Reflected in University President Bulk Email
Authors:
Ruoyan Kong,
Charles Chuankai Zhang,
Jin Kang,
Haiyi Zhu,
Joseph A. Konstan
Abstract:
E-mail ``Messages From the President'' to university students, staff, and faculty have long been used to keep campus communities aware of the latest policies, events, and news. But during the COVID-19 pandemic, as universities quickly closed facilities, sent students home, and canceled travel, these messages took on even greater importance. We report on a content analysis of bulk emails from diffe…
▽ More
E-mail ``Messages From the President'' to university students, staff, and faculty have long been used to keep campus communities aware of the latest policies, events, and news. But during the COVID-19 pandemic, as universities quickly closed facilities, sent students home, and canceled travel, these messages took on even greater importance. We report on a content analysis of bulk emails from different universities' presidents to their students and employees before and in three stages of the pandemic. We find that these messages change as universities move towards and through closure. During the pandemic, 1) presidential bulk emails tend to be more informative, positive, clearer than before; 2) they tend to use more personal and collective language; 3) university presidents tend to mention more local political leaders and fewer other university leaders. Our results can inform research on digital crisis communication and may be useful for researchers interested in automatically identifying crisis situations from communication streams.
△ Less
Submitted 20 January, 2024;
originally announced January 2024.
-
Interactive Content Diversity and User Exploration in Online Movie Recommenders: A Field Experiment
Authors:
Ruixuan Sun,
Avinash Akella,
Ruoyan Kong,
Moyan Zhou,
Joseph A. Konstan
Abstract:
Recommender systems often struggle to strike a balance between matching users' tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users' preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don't like, it is considered impersonal or ineffective. To better understand user…
▽ More
Recommender systems often struggle to strike a balance between matching users' tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users' preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don't like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different level of exploration control and users' subjective preferences on interfaces are more predictive of their satisfaction with the recommender.
△ Less
Submitted 23 September, 2023;
originally announced September 2023.
-
Organizational Bulk Email Systems: Their Role and Performance in Remote Work
Authors:
Ruoyan Kong,
Haiyi Zhu,
Joseph A. Konstan
Abstract:
The COVID-19 pandemic has forced many employees to work from home. Organizational bulk emails now play a critical role to reach employees with central information in this work-from-home environment. However, we know from our own recent work that organizational bulk email has problems: recipients fail to retain the bulk messages they received from the organization; recipients and senders have diffe…
▽ More
The COVID-19 pandemic has forced many employees to work from home. Organizational bulk emails now play a critical role to reach employees with central information in this work-from-home environment. However, we know from our own recent work that organizational bulk email has problems: recipients fail to retain the bulk messages they received from the organization; recipients and senders have different opinions on which bulk messages were important; and communicators lack technology support to better target and design messages. In this position paper, first we review the prior work on evaluating, designing, and prototyping organizational communication systems. Second we review our recent findings and some research techniques we found useful in studying organizational communication. Last we propose a research agenda to study organizational communications in remote work environment and suggest some key questions and potential study directions.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems
Authors:
Ruixuan Sun,
Ruoyan Kong,
Qiao Jin,
Joseph A. Konstan
Abstract:
In this study, we partition users by rating disposition - looking first at their percentage of negative ratings, and then at the general use of the rating scale. We hypothesize that users with different rating dispositions may use the recommender system differently and therefore the agreement with their past ratings may be less predictive of the future agreement.
We use data from a large movie r…
▽ More
In this study, we partition users by rating disposition - looking first at their percentage of negative ratings, and then at the general use of the rating scale. We hypothesize that users with different rating dispositions may use the recommender system differently and therefore the agreement with their past ratings may be less predictive of the future agreement.
We use data from a large movie rating website to explore whether users should be grouped by disposition, focusing on identifying their various rating distributions that may hurt recommender effectiveness. We find that such partitioning not only improves computational efficiency but also improves top-k performance and predictive accuracy. Though such effects are largest for the user-based KNN CF, smaller for item-based KNN CF, and smallest for latent factor algorithms such as SVD.
△ Less
Submitted 20 June, 2023;
originally announced June 2023.
-
Getting the Most from Eye-Tracking: User-Interaction Based Reading Region Estimation Dataset and Models
Authors:
Ruoyan Kong,
Ruixuan Sun,
Charles Chuankai Zhang,
Chen Chen,
Sneha Patri,
Gayathri Gajjela,
Joseph A. Konstan
Abstract:
A single digital newsletter usually contains many messages (regions). Users' reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users' interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time b…
▽ More
A single digital newsletter usually contains many messages (regions). Users' reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users' interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data.
With eye-tracking, we collected 200k ground-truth datapoints on participants reading news on browsers. Then we trained machine learning and deep learning models to predict message-level reading time based on user interactions like mouse position, scrolling, and clicking. We reached 27\% percentage error in reading time estimation with a two-tower neural network based on user interactions only, against the eye-tracking ground truth data, while the heuristic baselines have around 46\% percentage error. We also discovered the benefits of replacing per-session models with per-timestamp models, and adding user pattern features. We concluded with suggestions on developing message-level reading estimation techniques based on available data.
△ Less
Submitted 12 June, 2023;
originally announced June 2023.
-
Multi-Objective Personalization in Multi-Stakeholder Organizational Bulk E-mail: A Field Experiment
Authors:
Ruoyan Kong,
Chuankai Zhang,
Ruixuan Sun,
Vishnu Chhabra,
Tanushsrisai Nadimpalli,
Joseph A. Konstan
Abstract:
Bulk email is often used in organizations to communicate ``important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining emplo…
▽ More
Bulk email is often used in organizations to communicate ``important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees' positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals).
Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees' and the organization's preferences. We measured these designs' influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it.
We found that ``important-to-organization'' messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter's recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization's preferences conflicted. Finally, we discuss the design suggestions on personalization and recommendation for organizational bulk email.
△ Less
Submitted 17 July, 2023; v1 submitted 22 February, 2023;
originally announced February 2023.
-
Learning to Ignore: A Case Study of Organization-Wide Bulk Email Effectiveness
Authors:
Ruoyan Kong,
Haiyi Zhu,
Joseph A. Konstan
Abstract:
Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email s…
▽ More
Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email system is a multi-stakeholder problem including recipients, communicators, and the organization itself. We studied the effectiveness, practice, and assessments of the organizational bulk email system of a large university from multi-stakeholders' perspectives. We conducted a qualitative study with the university's communicators, recipients, and managers. We delved into the organizational bulk email's distributing mechanisms of the communicators, the reading behaviors of recipients, and the perspectives on emails' values of communicators, managers, and recipients. We found that the organizational bulk email system as a whole was strained, and communicators are caught in the middle of this multi-stakeholder problem. First, though the communicators had an interest in preserving the effectiveness of channels in reaching employees, they had high-level clients whose interests might outweigh judgment about whether a message deserves widespread circulation. Second, though communicators thought they were sending important information, recipients viewed most of the organizational bulk emails as not relevant to them. Third, this disagreement was amplified by the success metric used by communicators. They viewed their bulk emails as successful if they had a high open rate. But recipients often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general understood the challenge, they had a limited set of targeting and feedback tools to support their task.
△ Less
Submitted 6 February, 2021; v1 submitted 29 June, 2020;
originally announced June 2020.
-
Recommender Systems Notation: Proposed Common Notation for Teaching and Research
Authors:
Michael D. Ekstrand,
Joseph A. Konstan
Abstract:
As the field of recommender systems has developed, authors have used a myriad of notations for describing the mathematical workings of recommendation algorithms. These notations ap-pear in research papers, books, lecture notes, blog posts, and software documentation. The dis-ciplinary diversity of the field has not contributed to consistency in notation; scholars whose home base is in information…
▽ More
As the field of recommender systems has developed, authors have used a myriad of notations for describing the mathematical workings of recommendation algorithms. These notations ap-pear in research papers, books, lecture notes, blog posts, and software documentation. The dis-ciplinary diversity of the field has not contributed to consistency in notation; scholars whose home base is in information retrieval have different habits and expectations than those in ma-chine learning or human-computer interaction.
In the course of years of teaching and research on recommender systems, we have seen the val-ue in adopting a consistent notation across our work. This has been particularly highlighted in our development of the Recommender Systems MOOC on Coursera (Konstan et al. 2015), as we need to explain a wide variety of algorithms and our learners are not well-served by changing notation between algorithms.
In this paper, we describe the notation we have adopted in our work, along with its justification and some discussion of considered alternatives. We present this in hope that it will be useful to others writing and teaching about recommender systems. This notation has served us well for some time now, in research, online education, and traditional classroom instruction. We feel it is ready for broad use.
△ Less
Submitted 4 February, 2019;
originally announced February 2019.
