-
Not Even Nice Work If You Can Get It; A Longitudinal Study of Uber's Algorithmic Pay and Pricing
Authors:
Reuben Binns,
Jake Stein,
Siddhartha Datta,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Ride-sharing platforms like Uber market themselves as enabling `flexibility' for their workforce, meaning that drivers are expected to anticipate when and where the algorithm will allocate them jobs, and how well remunerated those jobs will be. In this work we describe our process of participatory action research with drivers and trade union organisers, culminating in a participatory audit of Uber…
▽ More
Ride-sharing platforms like Uber market themselves as enabling `flexibility' for their workforce, meaning that drivers are expected to anticipate when and where the algorithm will allocate them jobs, and how well remunerated those jobs will be. In this work we describe our process of participatory action research with drivers and trade union organisers, culminating in a participatory audit of Uber's algorithmic pay and work allocation, before and after the introduction of dynamic pricing. Through longitudinal analysis of 1.5 million trips from 258 drivers in the UK, we find that after dynamic pricing, pay has decreased, Uber's cut has increased, job allocation and pay is less predictable, inequality between drivers is increased, and drivers spend more time waiting for jobs. In addition to these findings, we provide methodological and theoretical contributions to algorithm auditing, gig work, and the emerging practice of worker data science.
△ Less
Submitted 18 June, 2025;
originally announced June 2025.
-
The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systems
Authors:
Sruthi Viswanathan,
Seray Ibrahim,
Ravi Shankar,
Reuben Binns,
Max Van Kleek,
Petr Slovak
Abstract:
Parenting brings emotional and physical challenges, from balancing work, childcare, and finances to coping with exhaustion and limited personal time. Yet, one in three parents never seek support. AI systems potentially offer stigma-free, accessible, and affordable solutions. Yet, user adoption often fails due to issues with explainability and reliability. To see if these issues could be solved usi…
▽ More
Parenting brings emotional and physical challenges, from balancing work, childcare, and finances to coping with exhaustion and limited personal time. Yet, one in three parents never seek support. AI systems potentially offer stigma-free, accessible, and affordable solutions. Yet, user adoption often fails due to issues with explainability and reliability. To see if these issues could be solved using a co-design approach, we developed and tested NurtureBot, a wellbeing support assistant for new parents. 32 parents co-designed the system through Asynchronous Remote Communities method, identifying the key challenge as achieving a "successful chat." As part of co-design, parents role-played as NurtureBot, rewriting its dialogues to improve user understanding, control, and outcomes. The refined prototype, featuring an Interaction Layer, was evaluated by 32 initial and 46 new parents, showing improved user experience and usability, with final CUQ score of 91.3/100, demonstrating successful interaction patterns. Our process revealed useful interaction design lessons for effective AI parenting support.
△ Less
Submitted 12 March, 2025; v1 submitted 2 November, 2024;
originally announced November 2024.
-
How Should We Support Designing Privacy-Friendly Apps for Children? Using a Research through Design Process to Understand Developers' Needs and Challenges
Authors:
Anirudh Ekambaranathan,
Jun Zhao,
Max Van Kleek
Abstract:
Mobile apps used by children often make use of harmful techniques, such as data tracking and targeted advertising. Previous research has suggested that developers face several systemic challenges in designing apps that prioritise children's best interests. To understand how developers can be better supported, we used a Research through Design (RtD) method to explore what the future of privacy-frie…
▽ More
Mobile apps used by children often make use of harmful techniques, such as data tracking and targeted advertising. Previous research has suggested that developers face several systemic challenges in designing apps that prioritise children's best interests. To understand how developers can be better supported, we used a Research through Design (RtD) method to explore what the future of privacy-friendly app development could look like. We performed an elicitation study with 20 children's app developers to understand their needs and requirements. We found a number of specific technical requirements from the participants about how they would like to be supported, such as having actionable transnational design guidelines and easy-to-use development libraries. However, participants were reluctant to adopt these design ideas in their development practices due to perceived financial risks associated with increased privacy in apps. To overcome this critical gap, participants formulated socio-technical requirements that extend to other stakeholders in the mobile industry, including parents and marketplaces. Our findings provide important immediate and long-term design opportunities for the HCI community, and indicate that support for changing app developers' practices must be designed in the context of their relationship with other stakeholders.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
We Are Not There Yet: The Implications of Insufficient Knowledge Management for Organisational Compliance
Authors:
Thomas Şerban von Davier,
Konrad Kollnig,
Reuben Binns,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Since GDPR went into effect in 2018, many other data protection and privacy regulations have been released. With the new regulation, there has been an associated increase in industry professionals focused on data protection and privacy. Building on related work showing the potential benefits of knowledge management in organisational compliance and privacy engineering, this paper presents the findi…
▽ More
Since GDPR went into effect in 2018, many other data protection and privacy regulations have been released. With the new regulation, there has been an associated increase in industry professionals focused on data protection and privacy. Building on related work showing the potential benefits of knowledge management in organisational compliance and privacy engineering, this paper presents the findings of an exploratory qualitative study with data protection officers and other privacy professionals. We found issues with knowledge management to be the underlying challenge of our participants' feedback. Our participants noted four categories of feedback: (1) a perceived disconnect between regulation and practice, (2) a general lack of clear job description, (3) the need for data protection and privacy to be involved at every level of an organisation, (4) knowledge management tools exist but are not used effectively. This paper questions what knowledge management or automation solutions may prove to be effective in establishing better computer-supported work environments.
△ Less
Submitted 6 May, 2023;
originally announced May 2023.
-
Computers as Bad Social Actors: Dark Patterns and Anti-Patterns in Interfaces that Act Socially
Authors:
Lize Alberts,
Ulrik Lyngs,
Max Van Kleek
Abstract:
Technologies increasingly mimic human-like social behaviours. Beyond prototypical conversational agents like chatbots, this also applies to basic automated systems like app notifications or self-checkout machines that address or 'talk to' users in everyday situations. Whilst early evidence suggests social cues may enhance user experience, we lack a good understanding of when, and why, their use ma…
▽ More
Technologies increasingly mimic human-like social behaviours. Beyond prototypical conversational agents like chatbots, this also applies to basic automated systems like app notifications or self-checkout machines that address or 'talk to' users in everyday situations. Whilst early evidence suggests social cues may enhance user experience, we lack a good understanding of when, and why, their use may be inappropriate. Building on a survey of English-speaking smartphone users (n=80), we conducted experience sampling, interview, and workshop studies (n=11) to elicit people's attitudes and preferences regarding how automated systems talk to them. We thematically analysed examples of phrasings/conduct participants disliked, the reasons they gave, and what they would prefer instead. One category of inappropriate behaviour we identified regards the use of social cues as tools for manipulation. We describe four unwanted tactics interfaces use: agents playing on users' emotions (e.g., guilt-tripping or coaxing them), being pushy, `mothering' users, or being passive-aggressive. Another category regards pragmatics: personal or situational factors that can make a seemingly friendly or helpful utterance come across as rude, tactless, or invasive. These include failing to account for relevant contextual particulars (e.g., embarrassing users in public); expressing obviously false personalised care; or treating a user in ways that they find inappropriate for the system's role or the nature of their relationship. We discuss these behaviours in terms of an emerging 'social' class of dark and anti-patterns. Drawing from participant recommendations, we offer suggestions for improving how interfaces treat people in interactions, including broader normative reflections on treating users respectfully.
△ Less
Submitted 5 January, 2024; v1 submitted 9 February, 2023;
originally announced February 2023.
-
Respect as a Lens for the Design of AI Systems
Authors:
William Seymour,
Max Van Kleek,
Reuben Binns,
Dave Murray-Rust
Abstract:
Critical examinations of AI systems often apply principles such as fairness, justice, accountability, and safety, which is reflected in AI regulations such as the EU AI Act. Are such principles sufficient to promote the design of systems that support human flourishing? Even if a system is in some sense fair, just, or 'safe', it can nonetheless be exploitative, coercive, inconvenient, or otherwise…
▽ More
Critical examinations of AI systems often apply principles such as fairness, justice, accountability, and safety, which is reflected in AI regulations such as the EU AI Act. Are such principles sufficient to promote the design of systems that support human flourishing? Even if a system is in some sense fair, just, or 'safe', it can nonetheless be exploitative, coercive, inconvenient, or otherwise conflict with cultural, individual, or social values. This paper proposes a dimension of interactional ethics thus far overlooked: the ways AI systems should treat human beings. For this purpose, we explore the philosophical concept of respect: if respect is something everyone needs and deserves, shouldn't technology aim to be respectful? Despite its intuitive simplicity, respect in philosophy is a complex concept with many disparate senses. Like fairness or justice, respect can characterise how people deserve to be treated; but rather than relating primarily to the distribution of benefits or punishments, respect relates to how people regard one another, and how this translates to perception, treatment, and behaviour. We explore respect broadly across several literatures, synthesising perspectives on respect from Kantian, post-Kantian, dramaturgical, and agential realist design perspectives with a goal of drawing together a view of what respect could mean for AI. In so doing, we identify ways that respect may guide us towards more sociable artefacts that ethically and inclusively honour and recognise humans using the rich social language that we have evolved to interact with one another every day.
△ Less
Submitted 15 June, 2022;
originally announced June 2022.
-
Goodbye Tracking? Impact of iOS App Tracking Transparency and Privacy Labels
Authors:
Konrad Kollnig,
Anastasia Shuba,
Max Van Kleek,
Reuben Binns,
Nigel Shadbolt
Abstract:
Tracking is a highly privacy-invasive data collection practice that has been ubiquitous in mobile apps for many years due to its role in supporting advertising-based revenue models. In response, Apple introduced two significant changes with iOS 14: App Tracking Transparency (ATT), a mandatory opt-in system for enabling tracking on iOS, and Privacy Nutrition Labels, which disclose what kinds of dat…
▽ More
Tracking is a highly privacy-invasive data collection practice that has been ubiquitous in mobile apps for many years due to its role in supporting advertising-based revenue models. In response, Apple introduced two significant changes with iOS 14: App Tracking Transparency (ATT), a mandatory opt-in system for enabling tracking on iOS, and Privacy Nutrition Labels, which disclose what kinds of data each app processes. So far, the impact of these changes on individual privacy and control has not been well understood. This paper addresses this gap by analysing two versions of 1,759 iOS apps from the UK App Store: one version from before iOS 14 and one that has been updated to comply with the new rules.
We find that Apple's new policies, as promised, prevent the collection of the Identifier for Advertisers (IDFA), an identifier for cross-app tracking. Smaller data brokers that engage in invasive data practices will now face higher challenges in tracking users - a positive development for privacy. However, the number of tracking libraries has roughly stayed the same in the studied apps. Many apps still collect device information that can be used to track users at a group level (cohort tracking) or identify individuals probabilistically (fingerprinting). We find real-world evidence of apps computing and agreeing on a fingerprinting-derived identifier through the use of server-side code, thereby violating Apple's policies. We find that Apple itself engages in some forms of tracking and exempts invasive data practices like first-party tracking and credit scoring. We also find that the new Privacy Nutrition Labels are sometimes inaccurate and misleading.
Overall, our findings suggest that, while tracking individual users is more difficult now, the changes reinforce existing market power of gatekeeper companies with access to large troves of first-party data and motivate a countermovement.
△ Less
Submitted 7 May, 2022; v1 submitted 7 April, 2022;
originally announced April 2022.
-
Before and after GDPR: tracking in mobile apps
Authors:
Konrad Kollnig,
Reuben Binns,
Max Van Kleek,
Ulrik Lyngs,
Jun Zhao,
Claudine Tinsman,
Nigel Shadbolt
Abstract:
Third-party tracking, the collection and sharing of behavioural data about individuals, is a significant and ubiquitous privacy threat in mobile apps. The EU General Data Protection Regulation (GDPR) was introduced in 2018 to protect personal data better, but there exists, thus far, limited empirical evidence about its efficacy. This paper studies tracking in nearly two million Android apps from b…
▽ More
Third-party tracking, the collection and sharing of behavioural data about individuals, is a significant and ubiquitous privacy threat in mobile apps. The EU General Data Protection Regulation (GDPR) was introduced in 2018 to protect personal data better, but there exists, thus far, limited empirical evidence about its efficacy. This paper studies tracking in nearly two million Android apps from before and after the introduction of the GDPR. Our analysis suggests that there has been limited change in the presence of third-party tracking in apps, and that the concentration of tracking capabilities among a few large gatekeeper companies persists. However, change might be imminent.
△ Less
Submitted 21 December, 2021;
originally announced December 2021.
-
"Money makes the world go around'': Identifying Barriers to Better Privacy in Children's Apps From Developers' Perspectives
Authors:
Anirudh Ekambaranathan,
Jun Zhao,
Max Van Kleek
Abstract:
The industry for children's apps is thriving at the cost of children's privacy: these apps routinely disclose children's data to multiple data trackers and ad networks. As children spend increasing time online, such exposure accumulates to long-term privacy risks. In this paper, we used a mixed-methods approach to investigate why this is happening and how developers might change their practices. W…
▽ More
The industry for children's apps is thriving at the cost of children's privacy: these apps routinely disclose children's data to multiple data trackers and ad networks. As children spend increasing time online, such exposure accumulates to long-term privacy risks. In this paper, we used a mixed-methods approach to investigate why this is happening and how developers might change their practices. We base our analysis against 5 leading data protection frameworks that set out requirements and recommendations for data collection in children's apps. To understand developers' perspectives and constraints, we conducted 134 surveys and 20 semi-structured interviews with popular Android children's app developers. Our analysis revealed that developers largely respect children's best interests; however, they have to make compromises due to limited monetisation options, perceived harmlessness of certain third-party libraries, and lack of availability of design guidelines. We identified concrete approaches and directions for future research to help overcome these barriers.
△ Less
Submitted 29 November, 2021;
originally announced November 2021.
-
Are iPhones Really Better for Privacy? Comparative Study of iOS and Android Apps
Authors:
Konrad Kollnig,
Anastasia Shuba,
Reuben Binns,
Max Van Kleek,
Nigel Shadbolt
Abstract:
While many studies have looked at privacy properties of the Android and Google Play app ecosystem, comparatively much less is known about iOS and the Apple App Store, the most widely used ecosystem in the US. At the same time, there is increasing competition around privacy between these smartphone operating system providers. In this paper, we present a study of 24k Android and iOS apps from 2020 a…
▽ More
While many studies have looked at privacy properties of the Android and Google Play app ecosystem, comparatively much less is known about iOS and the Apple App Store, the most widely used ecosystem in the US. At the same time, there is increasing competition around privacy between these smartphone operating system providers. In this paper, we present a study of 24k Android and iOS apps from 2020 along several dimensions relating to user privacy. We find that third-party tracking and the sharing of unique user identifiers was widespread in apps from both ecosystems, even in apps aimed at children. In the children's category, iOS apps tended to use fewer advertising-related tracking than their Android counterparts, but could more often access children's location. Across all studied apps, our study highlights widespread potential violations of US, EU and UK privacy law, including 1) the use of third-party tracking without user consent, 2) the lack of parental consent before sharing personally identifiable information (PII) with third-parties in children's apps, 3) the non-data-minimising configuration of tracking libraries, 4) the sending of personal data to countries without an adequate level of data protection, and 5) the continued absence of transparency around tracking, partly due to design decisions by Apple and Google. Overall, we find that neither platform is clearly better than the other for privacy across the dimensions we studied.
△ Less
Submitted 19 December, 2021; v1 submitted 28 September, 2021;
originally announced September 2021.
-
Protection or punishment? relating the design space of parental control apps and perceptions about them to support parenting for online safety
Authors:
Ge Wang,
Jun Zhao,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Parental control apps, which are mobile apps that allow parents to monitor and restrict their children's activities online, are becoming increasingly adopted by parents as a means of safeguarding their children's online safety. However, it is not clear whether these apps are always beneficial or effective in what they aim to do; for instance, the overuse of restriction and surveillance has been fo…
▽ More
Parental control apps, which are mobile apps that allow parents to monitor and restrict their children's activities online, are becoming increasingly adopted by parents as a means of safeguarding their children's online safety. However, it is not clear whether these apps are always beneficial or effective in what they aim to do; for instance, the overuse of restriction and surveillance has been found to undermine parent-child relationship and children's sense of autonomy. In this work, we investigate this gap, asking specifically: how might children's and parents' perceptions be related to how parental control features were designed? To investigate this question, we conducted an analysis of 58 top Android parental control apps designed for the purpose of promoting children's online safety, finding three major axes of variation in how key restriction and monitoring features were realised: granularity, feedback/transparency, and parent-child communications support. To relate these axes to perceived benefits and problems, we then analysed 3264 app reviews to identify references to aspects of the each of the axes above, to understand children's and parents' views of how such dimensions related to their experiences with these apps. Our findings led towards 1) an understanding of how parental control apps realise their functionalities differently along three axes of variation, 2) an analysis of exactly the ways that such variation influences children's and parents' perceptions, respectively of the usefulness or effectiveness of these apps, and finally 3) an identification of design recommendations and opportunities for future apps by contextualising our findings within existing digital parenting theories.
△ Less
Submitted 11 September, 2021;
originally announced September 2021.
-
Exploring Interactions Between Trust, Anthropomorphism, and Relationship Development in Voice Assistants
Authors:
William Seymour,
Max Van Kleek
Abstract:
Modern conversational agents such as Alexa and Google Assistant represent significant progress in speech recognition, natural language processing, and speech synthesis. But as these agents have grown more realistic, concerns have been raised over how their social nature might unconsciously shape our interactions with them. Through a survey of 500 voice assistant users, we explore whether users' re…
▽ More
Modern conversational agents such as Alexa and Google Assistant represent significant progress in speech recognition, natural language processing, and speech synthesis. But as these agents have grown more realistic, concerns have been raised over how their social nature might unconsciously shape our interactions with them. Through a survey of 500 voice assistant users, we explore whether users' relationships with their voice assistants can be quantified using the same metrics as social, interpersonal relationships; as well as if this correlates with how much they trust their devices and the extent to which they anthropomorphise them. Using Knapp's staircase model of human relationships, we find that not only can human-device interactions be modelled in this way, but also that relationship development with voice assistants correlates with increased trust and anthropomorphism.
△ Less
Submitted 4 August, 2021;
originally announced August 2021.
-
A Fait Accompli? An Empirical Study into the Absence of Consent to Third-Party Tracking in Android Apps
Authors:
Konrad Kollnig,
Reuben Binns,
Pierre Dewitte,
Max Van Kleek,
Ge Wang,
Daniel Omeiza,
Helena Webb,
Nigel Shadbolt
Abstract:
Third-party tracking allows companies to collect users' behavioural data and track their activity across digital devices. This can put deep insights into users' private lives into the hands of strangers, and often happens without users' awareness or explicit consent. EU and UK data protection law, however, requires consent, both 1) to access and store information on users' devices and 2) to legiti…
▽ More
Third-party tracking allows companies to collect users' behavioural data and track their activity across digital devices. This can put deep insights into users' private lives into the hands of strangers, and often happens without users' awareness or explicit consent. EU and UK data protection law, however, requires consent, both 1) to access and store information on users' devices and 2) to legitimate the processing of personal data as part of third-party tracking, as we analyse in this paper.
This paper further investigates whether and to what extent consent is implemented in mobile apps. First, we analyse a representative sample of apps from the Google Play Store. We find that most apps engage in third-party tracking, but few obtained consent before doing so, indicating potentially widespread violations of EU and UK privacy law. Second, we examine the most common third-party tracking libraries in detail. While most acknowledge that they rely on app developers to obtain consent on their behalf, they typically fail to put in place robust measures to ensure this: disclosure of consent requirements is limited; default consent implementations are lacking; and compliance guidance is difficult to find, hard to read, and poorly maintained.
△ Less
Submitted 18 June, 2021; v1 submitted 17 June, 2021;
originally announced June 2021.
-
I Want My App That Way: Reclaiming Sovereignty Over Personal Devices
Authors:
Konrad Kollnig,
Siddhartha Datta,
Max Van Kleek
Abstract:
Dark patterns in mobile apps take advantage of cognitive biases of end-users and can have detrimental effects on people's lives. Despite growing research in identifying remedies for dark patterns and established solutions for desktop browsers, there exists no established methodology to reduce dark patterns in mobile apps. Our work introduces GreaseDroid, a community-driven app modification framewo…
▽ More
Dark patterns in mobile apps take advantage of cognitive biases of end-users and can have detrimental effects on people's lives. Despite growing research in identifying remedies for dark patterns and established solutions for desktop browsers, there exists no established methodology to reduce dark patterns in mobile apps. Our work introduces GreaseDroid, a community-driven app modification framework enabling non-expert users to disable dark patterns in apps selectively.
△ Less
Submitted 23 February, 2021;
originally announced February 2021.
-
Exploring Design and Governance Challenges in the Development of Privacy-Preserving Computation
Authors:
Nitin Agrawal,
Reuben Binns,
Max Van Kleek,
Kim Laine,
Nigel Shadbolt
Abstract:
Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We i…
▽ More
Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We interviewed researchers, developers, industry leaders, policymakers, and designers involved in their deployment to explore motivations, expectations, perceived opportunities and barriers to adoption. This provided insight into several pertinent challenges facing the adoption of these technologies, including: how they might make a nebulous concept like privacy computationally tractable; how to make them more usable by developers; and how they could be explained and made accountable to stakeholders and wider society. We conclude with implications for the development, deployment, and responsible governance of these privacy-preserving computation techniques.
△ Less
Submitted 20 January, 2021;
originally announced January 2021.
-
COVID-19 what have we learned? The rise of social machines and connected devices in pandemic management following the concepts of predictive, preventive and personalised medicine
Authors:
Petar Radanliev,
David De Roure,
Rob Walton,
Max Van Kleek,
Rafael Mantilla Montalvo,
Omar Santos,
LaTreall Maddox,
Stacy Cannady
Abstract:
A comprehensive bibliographic review with R statistical methods of the COVID pandemic in PubMed literature and Web of Science Core Collection, supported with Google Scholar search. In addition, a case study review of emerging new approaches in different regions, using medical literature, academic literature, news articles and other reliable data sources. Public responses of mistrust about privacy…
▽ More
A comprehensive bibliographic review with R statistical methods of the COVID pandemic in PubMed literature and Web of Science Core Collection, supported with Google Scholar search. In addition, a case study review of emerging new approaches in different regions, using medical literature, academic literature, news articles and other reliable data sources. Public responses of mistrust about privacy data misuse differ across countries, depending on the chosen public communication strategy.
△ Less
Submitted 23 November, 2020; v1 submitted 12 September, 2020;
originally announced September 2020.
-
Digitalization of COVID-19 pandemic management and cyber risk from connected systems
Authors:
Petar Radanliev,
David De Roure,
Max Van Kleek
Abstract:
What makes cyber risks arising from connected systems challenging during the management of a pandemic? Assuming that a variety of cyber-physical systems are already operational-collecting, analyzing, and acting on data autonomously-what risks might arise in their application to pandemic management? We already have these systems operational, collecting, and analyzing data autonomously, so how would…
▽ More
What makes cyber risks arising from connected systems challenging during the management of a pandemic? Assuming that a variety of cyber-physical systems are already operational-collecting, analyzing, and acting on data autonomously-what risks might arise in their application to pandemic management? We already have these systems operational, collecting, and analyzing data autonomously, so how would a pandemic monitoring app be different or riskier? In this review article, we discuss the digitalization of COVID-19 pandemic management and cyber risk from connected systems.
△ Less
Submitted 25 May, 2020;
originally announced May 2020.
-
Design of a dynamic and self adapting system, supported with artificial intelligence, machine learning and real time intelligence for predictive cyber risk analytics in extreme environments, cyber risk in the colonisation of Mars
Authors:
Petar Radanliev,
David De Roure,
Kevin Page,
Max Van Kleek,
Omar Santos,
La Treall Maddox,
Pete Burnap,
Eirini Anthi,
Carsten Maple
Abstract:
Multiple governmental agencies and private organisations have made commitments for the colonisation of Mars. Such colonisation requires complex systems and infrastructure that could be very costly to repair or replace in cases of cyber attacks. This paper surveys deep learning algorithms, IoT cyber security and risk models, and established mathematical formulas to identify the best approach for de…
▽ More
Multiple governmental agencies and private organisations have made commitments for the colonisation of Mars. Such colonisation requires complex systems and infrastructure that could be very costly to repair or replace in cases of cyber attacks. This paper surveys deep learning algorithms, IoT cyber security and risk models, and established mathematical formulas to identify the best approach for developing a dynamic and self adapting system for predictive cyber risk analytics supported with Artificial Intelligence and Machine Learning and real time intelligence in edge computing. The paper presents a new mathematical approach for integrating concepts for cognition engine design, edge computing and Artificial Intelligence and Machine Learning to automate anomaly detection. This engine instigates a step change by applying Artificial Intelligence and Machine Learning embedded at the edge of IoT networks, to deliver safe and functional real time intelligence for predictive cyber risk analytics. This will enhance capacities for risk analytics and assists in the creation of a comprehensive and systematic understanding of the opportunities and threats that arise when edge computing nodes are deployed, and when Artificial Intelligence and Machine Learning technologies are migrated to the periphery of the internet and into local IoT networks.
△ Less
Submitted 11 February, 2021; v1 submitted 19 May, 2020;
originally announced May 2020.
-
What country, university or research institute, performed the best on COVID-19? Bibliometric analysis of scientific literature
Authors:
Petar Radanliev,
David De Roure,
Rob Walton,
Max Van Kleek,
Omar Santos,
La Treall Maddox
Abstract:
In this article, we conduct data mining to discover the countries, universities and companies, produced or collaborated the most research on Covid-19 since the pandemic started. We present some interesting findings, but despite analysing all available records on COVID-19 from the Web of Science Core Collection, we failed to reach any significant conclusions on how the world responded to the COVID-…
▽ More
In this article, we conduct data mining to discover the countries, universities and companies, produced or collaborated the most research on Covid-19 since the pandemic started. We present some interesting findings, but despite analysing all available records on COVID-19 from the Web of Science Core Collection, we failed to reach any significant conclusions on how the world responded to the COVID-19 pandemic. Therefore, we increased our analysis to include all available data records on pandemics and epidemics from 1900 to 2020. We discover some interesting results on countries, universities and companies, that produced collaborated most the most in research on pandemic and epidemics. Then we compared the results with the analysing on COVID-19 data records. This has created some interesting findings that are explained and graphically visualised in the article.
△ Less
Submitted 16 July, 2022; v1 submitted 19 May, 2020;
originally announced May 2020.
-
Strangers in the Room: Unpacking Perceptions of 'Smartness' and Related Ethical Concerns in the Home
Authors:
William Seymour,
Reuben Binns,
Petr Slovak,
Max Van Kleek,
Nigel Shadbolt
Abstract:
The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in gen…
▽ More
The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in general. This paper attempts to bridge these approaches by asking what features of smart devices users consider as rendering them 'smart' and how these relate to ethical concerns. Through a multimethod investigation including surveys with smart device users (n=120) and semi-structured interviews (n=15), we identify and describe eight types of smartness and explore how they engender a variety of ethical concerns including privacy, autonomy, and disruption of the social order. We argue that this middle ground, between concerns arising from particular devices and more abstract ethical concepts, can better anticipate potential ethical concerns regarding smart devices.
△ Less
Submitted 1 May, 2020;
originally announced May 2020.
-
Super-reflective Data: Speculative Imaginings of a World Where Data Works for People
Authors:
Max Van Kleek
Abstract:
It's the year 2020, and every space and place on- and off-line has been augmented with digital things that observe, record, transmit, and compute, for the purposes of recording endless data traces of what is happening in the world. Individually, these things (and the invisible services the power them) have reached considerable sophistication in their ability to analyse and dissect such observation…
▽ More
It's the year 2020, and every space and place on- and off-line has been augmented with digital things that observe, record, transmit, and compute, for the purposes of recording endless data traces of what is happening in the world. Individually, these things (and the invisible services the power them) have reached considerable sophistication in their ability to analyse and dissect such observations, turning streams of audio and video into informative data fragments. Yet somehow, individuals as end-users of platforms and services have not seen the full potential of such data. In this speculative paper, we propose two hypothetical mini scenarios different from our current digital world. In the former, instead of hoarding it, data controllers turn captured data over to those who need it as quickly as possible, working together to combine, validate, and refine it for maximum usefulness. This simultaneously addresses the data fragmentation and privacy problem, by handing over long-term data governance to those that value it the most In the latter, we discuss ethical dilemmas using the long-term use of such rich data and its tendency to cause people to relentlessly optimise.
△ Less
Submitted 10 March, 2020;
originally announced March 2020.
-
Does Siri Have a Soul? Exploring Voice Assistants Through Shinto Design Fictions
Authors:
William Seymour,
Max Van Kleek
Abstract:
It can be difficult to critically reflect on technology that has become part of everyday rituals and routines. To combat this, speculative and fictional approaches have previously been used by HCI to decontextualise the familiar and imagine alternatives. In this work we turn to Japanese Shinto narratives as a way to defamiliarise voice assistants, inspired by the similarities between how assistant…
▽ More
It can be difficult to critically reflect on technology that has become part of everyday rituals and routines. To combat this, speculative and fictional approaches have previously been used by HCI to decontextualise the familiar and imagine alternatives. In this work we turn to Japanese Shinto narratives as a way to defamiliarise voice assistants, inspired by the similarities between how assistants appear to 'inhabit' objects similarly to kami. Describing an alternate future where assistant presences live inside objects, this approach foregrounds some of the phenomenological quirks that can otherwise easily become lost. Divorced from the reality of daily life, this approach allows us to reevaluate some of the common interactions and design patterns that are common in the virtual assistants of the present.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Informing the Design of Privacy-Empowering Tools for the Connected Home
Authors:
William Seymour,
Martin J. Kraemer,
Reuben Binns,
Max Van Kleek
Abstract:
Connected devices in the home represent a potentially grave new privacy threat due to their unfettered access to the most personal spaces in people's lives. Prior work has shown that despite concerns about such devices, people often lack sufficient awareness, understanding, or means of taking effective action. To explore the potential for new tools that support such needs directly we developed Are…
▽ More
Connected devices in the home represent a potentially grave new privacy threat due to their unfettered access to the most personal spaces in people's lives. Prior work has shown that despite concerns about such devices, people often lack sufficient awareness, understanding, or means of taking effective action. To explore the potential for new tools that support such needs directly we developed Aretha, a privacy assistant technology probe that combines a network disaggregator, personal tutor, and firewall, to empower end-users with both the knowledge and mechanisms to control disclosures from their homes. We deployed Aretha in three households over six weeks, with the aim of understanding how this combination of capabilities might enable users to gain awareness of data disclosures by their devices, form educated privacy preferences, and to block unwanted data flows. The probe, with its novel affordances-and its limitations-prompted users to co-adapt, finding new control mechanisms and suggesting new approaches to address the challenge of regaining privacy in the connected home.
△ Less
Submitted 24 January, 2020;
originally announced January 2020.
-
'I Just Want to Hack Myself to Not Get Distracted': Evaluating Design Interventions for Self-Control on Facebook
Authors:
Ulrik Lyngs,
Kai Lukoff,
Petr Slovak,
William Seymour,
Helena Webb,
Marina Jirotka,
Jun Zhao,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived…
▽ More
Beyond being the world's largest social network, Facebook is for many also one of its greatest sources of digital distraction. For students, problematic use has been associated with negative effects on academic achievement and general wellbeing. To understand what strategies could help users regain control, we investigated how simple interventions to the Facebook UI affect behaviour and perceived control. We assigned 58 university students to one of three interventions: goal reminders, removed newsfeed, or white background (control). We logged use for 6 weeks, applied interventions in the middle weeks, and administered fortnightly surveys. Both goal reminders and removed newsfeed helped participants stay on task and avoid distraction. However, goal reminders were often annoying, and removing the newsfeed made some fear missing out on information. Our findings point to future interventions such as controls for adjusting types and amount of available information, and flexible blocking which matches individual definitions of 'distraction'.
△ Less
Submitted 20 May, 2020; v1 submitted 13 January, 2020;
originally announced January 2020.
-
Informing The Future of Data Protection in Smart Homes
Authors:
Martin J Kraemer,
William Seymour,
Reuben Binns,
Max Van Kleek,
Ivan Flechais
Abstract:
Recent changes to data protection regulation, particularly in Europe, are changing the design landscape for smart devices, requiring new design techniques to ensure that devices are able to adequately protect users' data. A particularly interesting space in which to explore and address these challenges is the smart home, which presents a multitude of difficult social and technical problems in an i…
▽ More
Recent changes to data protection regulation, particularly in Europe, are changing the design landscape for smart devices, requiring new design techniques to ensure that devices are able to adequately protect users' data. A particularly interesting space in which to explore and address these challenges is the smart home, which presents a multitude of difficult social and technical problems in an intimate and highly private context. This position paper outlines the motivation and research approach of a new project aiming to inform the future of data protection by design and by default in smart homes through a combination of ethnography and speculative design.
△ Less
Submitted 17 June, 2019;
originally announced October 2019.
-
Dynamic real-time risk analytics of uncontrollable states in complex internet of things systems, cyber risk at the edge
Authors:
Petar Radanliev,
David De Roure,
Max Van Kleek,
Uchenna Ani,
Pete Burnap,
Eirini Anthi,
Jason R. C. Nurse,
Omar Santos,
Rafael Mantilla Montalvo,
LaTreall Maddox
Abstract:
The Internet of Things (IoT) triggers new types of cyber risks. Therefore, the integration of new IoT devices and services requires a self-assessment of IoT cyber security posture. By security posture this article refers to the cybersecurity strength of an organisation to predict, prevent and respond to cyberthreats. At present, there is a gap in the state of the art, because there are no self-ass…
▽ More
The Internet of Things (IoT) triggers new types of cyber risks. Therefore, the integration of new IoT devices and services requires a self-assessment of IoT cyber security posture. By security posture this article refers to the cybersecurity strength of an organisation to predict, prevent and respond to cyberthreats. At present, there is a gap in the state of the art, because there are no self-assessment methods for quantifying IoT cyber risk posture. To address this gap, an empirical analysis is performed of 12 cyber risk assessment approaches. The results and the main findings from the analysis is presented as the current and a target risk state for IoT systems, followed by conclusions and recommendations on a transformation roadmap, describing how IoT systems can achieve the target state with a new goal-oriented dependency model. By target state, we refer to the cyber security target that matches the generic security requirements of an organisation. The research paper studies and adapts four alternatives for IoT risk assessment and identifies the goal-oriented dependency modelling as a dominant approach among the risk assessment models studied. The new goal-oriented dependency model in this article enables the assessment of uncontrollable risk states in complex IoT systems and can be used for a quantitative self-assessment of IoT cyber risk posture.
△ Less
Submitted 23 November, 2020; v1 submitted 12 March, 2019;
originally announced March 2019.
-
Artificial intelligence in cyber physical systems
Authors:
Petar Radanliev,
David De Roure,
Max Van Kleek,
Omar Santos,
Uchenna Ani
Abstract:
This article conducts a literature review of current and future challenges in the use of artificial intelligence (AI) in cyber physical systems. The literature review is focused on identifying a conceptual framework for increasing resilience with AI through automation supporting both, a technical and human level. The methodology applied resembled a literature review and taxonomic analysis of compl…
▽ More
This article conducts a literature review of current and future challenges in the use of artificial intelligence (AI) in cyber physical systems. The literature review is focused on identifying a conceptual framework for increasing resilience with AI through automation supporting both, a technical and human level. The methodology applied resembled a literature review and taxonomic analysis of complex internet of things (IoT) interconnected and coupled cyber physical systems. There is an increased attention on propositions on models, infrastructures and frameworks of IoT in both academic and technical papers. These reports and publications frequently represent a juxtaposition of other related systems and technologies (e.g. Industrial Internet of Things, Cyber Physical Systems, Industry 4.0 etc.). We review academic and industry papers published between 2010 and 2020. The results determine a new hierarchical cascading conceptual framework for analysing the evolution of AI decision-making in cyber physical systems. We argue that such evolution is inevitable and autonomous because of the increased integration of connected devices (IoT) in cyber physical systems. To support this argument, taxonomic methodology is adapted and applied for transparency and justifications of concepts selection decisions through building summary maps that are applied for designing the hierarchical cascading conceptual framework.
△ Less
Submitted 12 September, 2020; v1 submitted 11 March, 2019;
originally announced March 2019.
-
Self-Control in Cyberspace: Applying Dual Systems Theory to a Review of Digital Self-Control Tools
Authors:
Ulrik Lyngs,
Kai Lukoff,
Petr Slovak,
Reuben Binns,
Adam Slack,
Michael Inzlicht,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Many people struggle to control their use of digital devices. However, our understanding of the design mechanisms that support user self-control remains limited. In this paper, we make two contributions to HCI research in this space: first, we analyse 367 apps and browser extensions from the Google Play, Chrome Web, and Apple App stores to identify common core design features and intervention stra…
▽ More
Many people struggle to control their use of digital devices. However, our understanding of the design mechanisms that support user self-control remains limited. In this paper, we make two contributions to HCI research in this space: first, we analyse 367 apps and browser extensions from the Google Play, Chrome Web, and Apple App stores to identify common core design features and intervention strategies afforded by current tools for digital self-control. Second, we adapt and apply an integrative dual systems model of self-regulation as a framework for organising and evaluating the design features found. Our analysis aims to help the design of better tools in two ways: (i) by identifying how, through a well-established model of self-regulation, current tools overlap and differ in how they support self-control; and (ii) by using the model to reveal underexplored cognitive mechanisms that could aid the design of new tools.
△ Less
Submitted 31 January, 2019;
originally announced February 2019.
-
Third Party Tracking in the Mobile Ecosystem
Authors:
Reuben Binns,
Ulrik Lyngs,
Max Van Kleek,
Jun Zhao,
Timothy Libert,
Nigel Shadbolt
Abstract:
Third party tracking allows companies to identify users and track their behaviour across multiple digital services. This paper presents an empirical study of the prevalence of third-party trackers on 959,000 apps from the US and UK Google Play stores. We find that most apps contain third party tracking, and the distribution of trackers is long-tailed with several highly dominant trackers accountin…
▽ More
Third party tracking allows companies to identify users and track their behaviour across multiple digital services. This paper presents an empirical study of the prevalence of third-party trackers on 959,000 apps from the US and UK Google Play stores. We find that most apps contain third party tracking, and the distribution of trackers is long-tailed with several highly dominant trackers accounting for a large portion of the coverage. The extent of tracking also differs between categories of apps; in particular, news apps and apps targeted at children appear to be amongst the worst in terms of the number of third party trackers associated with them. Third party tracking is also revealed to be a highly trans-national phenomenon, with many trackers operating in jurisdictions outside the EU. Based on these findings, we draw out some significant legal compliance challenges facing the tracking industry.
△ Less
Submitted 18 October, 2018; v1 submitted 10 April, 2018;
originally announced April 2018.
-
Some HCI Priorities for GDPR-Compliant Machine Learning
Authors:
Michael Veale,
Reuben Binns,
Max Van Kleek
Abstract:
In this short paper, we consider the roles of HCI in enabling the better governance of consequential machine learning systems using the rights and obligations laid out in the recent 2016 EU General Data Protection Regulation (GDPR)---a law which involves heavy interaction with people and systems. Focussing on those areas that relate to algorithmic systems in society, we propose roles for HCI in le…
▽ More
In this short paper, we consider the roles of HCI in enabling the better governance of consequential machine learning systems using the rights and obligations laid out in the recent 2016 EU General Data Protection Regulation (GDPR)---a law which involves heavy interaction with people and systems. Focussing on those areas that relate to algorithmic systems in society, we propose roles for HCI in legal contexts in relation to fairness, bias and discrimination; data protection by design; data protection impact assessments; transparency and explanations; the mitigation and understanding of automation bias; and the communication of envisaged consequences of processing.
△ Less
Submitted 16 March, 2018;
originally announced March 2018.
-
"So, Tell Me What Users Want, What They Really, Really Want!"
Authors:
Ulrik Lyngs,
Reuben Binns,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Equating users' true needs and desires with behavioural measures of 'engagement' is problematic. However, good metrics of 'true preferences' are difficult to define, as cognitive biases make people's preferences change with context and exhibit inconsistencies over time. Yet, HCI research often glosses over the philosophical and theoretical depth of what it means to infer what users really want. In…
▽ More
Equating users' true needs and desires with behavioural measures of 'engagement' is problematic. However, good metrics of 'true preferences' are difficult to define, as cognitive biases make people's preferences change with context and exhibit inconsistencies over time. Yet, HCI research often glosses over the philosophical and theoretical depth of what it means to infer what users really want. In this paper, we present an alternative yet very real discussion of this issue, via a fictive dialogue between senior executives in a tech company aimed at helping people live the life they `really' want to live. How will the designers settle on a metric for their product to optimise?
△ Less
Submitted 6 March, 2018;
originally announced March 2018.
-
Measuring third party tracker power across web and mobile
Authors:
Reuben Binns,
Jun Zhao,
Max Van Kleek,
Nigel Shadbolt
Abstract:
Third-party networks collect vast amounts of data about users via web sites and mobile applications. Consolidations among tracker companies can significantly increase their individual tracking capabilities, prompting scrutiny by competition regulators. Traditional measures of market share, based on revenue or sales, fail to represent the tracking capability of a tracker, especially if it spans bot…
▽ More
Third-party networks collect vast amounts of data about users via web sites and mobile applications. Consolidations among tracker companies can significantly increase their individual tracking capabilities, prompting scrutiny by competition regulators. Traditional measures of market share, based on revenue or sales, fail to represent the tracking capability of a tracker, especially if it spans both web and mobile. This paper proposes a new approach to measure the concentration of tracking capability, based on the reach of a tracker on popular websites and apps. Our results reveal that tracker prominence and parent-subsidiary relationships have significant impact on accurately measuring concentration.
△ Less
Submitted 7 February, 2018;
originally announced February 2018.
-
Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
Authors:
Michael Veale,
Max Van Kleek,
Reuben Binns
Abstract:
Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions---like taxation, justice, and child protection---are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The res…
▽ More
Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions---like taxation, justice, and child protection---are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning---absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.
△ Less
Submitted 3 February, 2018;
originally announced February 2018.
-
'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
Authors:
Reuben Binns,
Max Van Kleek,
Michael Veale,
Ulrik Lyngs,
Jun Zhao,
Nigel Shadbolt
Abstract:
Data-driven decision-making consequential to individuals raises important questions of accountability and justice. Indeed, European law provides individuals limited rights to 'meaningful information about the logic' behind significant, autonomous decisions such as loan approvals, insurance quotes, and CV filtering. We undertake three experimental studies examining people's perceptions of justice i…
▽ More
Data-driven decision-making consequential to individuals raises important questions of accountability and justice. Indeed, European law provides individuals limited rights to 'meaningful information about the logic' behind significant, autonomous decisions such as loan approvals, insurance quotes, and CV filtering. We undertake three experimental studies examining people's perceptions of justice in algorithmic decision-making under different scenarios and explanation styles. Dimensions of justice previously observed in response to human decision-making appear similarly engaged in response to algorithmic decisions. Qualitative analysis identified several concerns and heuristics involved in justice perceptions including arbitrariness, generalisation, and (in)dignity. Quantitative analysis indicates that explanation styles primarily matter to justice perceptions only when subjects are exposed to multiple different styles---under repeated exposure of one style, scenario effects obscure any explanation effects. Our results suggests there may be no 'best' approach to explaining algorithmic decisions, and that reflection on their automated nature both implicates and mitigates justice dimensions.
△ Less
Submitted 31 January, 2018;
originally announced January 2018.
-
Like trainer, like bot? Inheritance of bias in algorithmic content moderation
Authors:
Reuben Binns,
Michael Veale,
Max Van Kleek,
Nigel Shadbolt
Abstract:
The internet has become a central medium through which `networked publics' express their opinions and engage in debate. Offensive comments and personal attacks can inhibit participation in these spaces. Automated content moderation aims to overcome this problem using machine learning classifiers trained on large corpora of texts manually annotated for offence. While such systems could help encoura…
▽ More
The internet has become a central medium through which `networked publics' express their opinions and engage in debate. Offensive comments and personal attacks can inhibit participation in these spaces. Automated content moderation aims to overcome this problem using machine learning classifiers trained on large corpora of texts manually annotated for offence. While such systems could help encourage more civil debate, they must navigate inherently normatively contestable boundaries, and are subject to the idiosyncratic norms of the human raters who provide the training data. An important objective for platforms implementing such measures might be to ensure that they are not unduly biased towards or against particular norms of offence. This paper provides some exploratory methods by which the normative biases of algorithmic content moderation systems can be measured, by way of a case study using an existing dataset of comments labelled for offence. We train classifiers on comments labelled by different demographic subsets (men and women) to understand how differences in conceptions of offence between these groups might affect the performance of the resulting models on various test sets. We conclude by discussing some of the ethical choices facing the implementers of algorithmic moderation systems, given various desired levels of diversity of viewpoints amongst discussion participants.
△ Less
Submitted 5 July, 2017;
originally announced July 2017.
-
Collective Intelligence in Citizen Science -- A Study of Performers and Talkers
Authors:
Ramine Tinati,
Elena Simperl,
Markus Luczak-Roesch,
Max Van Kleek,
Nigel Shadbolt
Abstract:
The recent emergence of online citizen science is illustrative of an efficient and effective means to harness the crowd in order to achieve a range of scientific discoveries. Fundamentally, citizen science projects draw upon crowds of non-expert volunteers to complete short Tasks, which can vary in domain and complexity. However, unlike most human-computational systems, participants in these syste…
▽ More
The recent emergence of online citizen science is illustrative of an efficient and effective means to harness the crowd in order to achieve a range of scientific discoveries. Fundamentally, citizen science projects draw upon crowds of non-expert volunteers to complete short Tasks, which can vary in domain and complexity. However, unlike most human-computational systems, participants in these systems, the `citizen scientists' are volunteers, whereby no incentives, financial or otherwise, are offered. Furthermore, encouraged by citizen science platforms such as Zooniverse, online communities have emerged, providing them with an environment to discuss, share ideas, and solve problems. In fact, it is the result of these forums that has enabled a number of scientific discoveries to be made. In this paper we explore the phenomenon of collective intelligence via the relationship between the activities of online citizen science communities and the discovery of scientific knowledge. We perform a cross-project analysis of ten Zooniverse citizen science projects and analyse the behaviour of users with regards to their Task completion activity and participation in discussion and discover collective behaviour amongst highly active users. Whilst our findings have implications for future citizen science design, we also consider the wider implications for understanding collective intelligence research in general.
△ Less
Submitted 29 June, 2014;
originally announced June 2014.
