-
"This could save us months of work" -- Use Cases of AI and Automation Support in Investigative Journalism
Authors:
Besjon Cifliku,
Hendrik Heuer
Abstract:
As the capabilities of Large Language Models (LLMs) expand, more researchers are studying their adoption in newsrooms. However, much of the research focus remains broad and does not address the specific technical needs of investigative journalists. Therefore, this paper presents several applied use cases where automation and AI intersect with investigative journalism. We conducted a within-subject…
▽ More
As the capabilities of Large Language Models (LLMs) expand, more researchers are studying their adoption in newsrooms. However, much of the research focus remains broad and does not address the specific technical needs of investigative journalists. Therefore, this paper presents several applied use cases where automation and AI intersect with investigative journalism. We conducted a within-subjects user study with eight investigative journalists. In interviews, we elicited practical use cases using a speculative design approach by having journalists react to a prototype of a system that combines LLMs and Programming-by-Demonstration (PbD) to simplify data collection on numerous websites. Based on user reports, we classified the journalistic processes into data collecting and reporting. Participants indicated they utilize automation to handle repetitive tasks like content monitoring, web scraping, summarization, and preliminary data exploration. Following these insights, we provide guidelines on how investigative journalism can benefit from AI and automation.
△ Less
Submitted 20 March, 2025;
originally announced March 2025.
-
Social Media for Activists: Reimagining Safety, Content Presentation, and Workflows
Authors:
Anna Ricarda Luther,
Hendrik Heuer,
Stephanie Geise,
Sebastian Haunss,
Andreas Breiter
Abstract:
Social media is central to activists, who use it internally for coordination and externally to reach supporters and the public. To date, the HCI community has not explored activists' perspectives on future social media platforms. In interviews with 14 activists from an environmental and a queer-feminist movement in Germany, we identify activists' needs and feature requests for future social media…
▽ More
Social media is central to activists, who use it internally for coordination and externally to reach supporters and the public. To date, the HCI community has not explored activists' perspectives on future social media platforms. In interviews with 14 activists from an environmental and a queer-feminist movement in Germany, we identify activists' needs and feature requests for future social media platforms. The key finding is that on- and offline safety is their main need. Based on this, we make concrete proposals to improve safety measures. Increased control over content presentation and tools to streamline activist workflows are also central to activists. We make concrete design and research recommendations on how social media platforms and the HCI community can contribute to improved safety and content presentation, and how activists themselves can reduce their workload.
△ Less
Submitted 20 March, 2025;
originally announced March 2025.
-
Social Media Journeys -- Mapping Platform Migration
Authors:
Artur Solomonik,
Hendrik Heuer
Abstract:
As people engage with the social media landscape, popular platforms rise and fall. As current research uncovers the experiences people have on various platforms, rarely do we engage with the sociotechnical migration processes when joining and leaving them. In this paper, we asked 32 visitors of a science communication festival to draw out artifacts that we call Social Media Journey Maps about the…
▽ More
As people engage with the social media landscape, popular platforms rise and fall. As current research uncovers the experiences people have on various platforms, rarely do we engage with the sociotechnical migration processes when joining and leaving them. In this paper, we asked 32 visitors of a science communication festival to draw out artifacts that we call Social Media Journey Maps about the social media platforms they frequented, and why. By combining qualitative content analysis with a graph representation of Social Media Journeys, we present how social media migration processes are motivated by the interplay of environmental and platform factors. We find that peer-driven popularity, the timing of feature adoption, and personal perceptions of migration causes - such as security - shape individuals' reasoning for migrating between social media platforms. With this work, we aim to pave the way for future social media platforms that foster meaningful and enriching online experiences for users.
△ Less
Submitted 18 March, 2025; v1 submitted 17 March, 2025;
originally announced March 2025.
-
Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations
Authors:
David Hartmann,
Amin Oueslati,
Dimitri Staufer,
Lena Pohlmann,
Simon Munzert,
Hendrik Heuer
Abstract:
Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework,…
▽ More
Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework, we systematically evaluate five widely used commercial content moderation APIs. Analyzing five million queries based on four datasets, we find that APIs frequently rely on group identity terms, such as ``black'', to predict hate speech. While OpenAI's and Amazon's services perform slightly better, all providers under-moderate implicit hate speech, which uses codified messages, especially against LGBTQIA+ individuals. Simultaneously, they over-moderate counter-speech, reclaimed slurs and content related to Black, LGBTQIA+, Jewish, and Muslim people. We recommend that API providers offer better guidance on API implementation and threshold setting and more transparency on their APIs' limitations.
Warning: This paper contains offensive and hateful terms and concepts. We have chosen to reproduce these terms for reasons of transparency.
△ Less
Submitted 3 March, 2025;
originally announced March 2025.
-
The Phase Model of Misinformation Interventions
Authors:
Hendrik Heuer
Abstract:
Misinformation is a challenging problem. This paper provides the first systematic interdisciplinary investigation of technical and non-technical interventions against misinformation. It combines interviews and a survey to understand which interventions are accepted across academic disciplines and approved by misinformation experts. Four interventions are supported by more than two in three misinfo…
▽ More
Misinformation is a challenging problem. This paper provides the first systematic interdisciplinary investigation of technical and non-technical interventions against misinformation. It combines interviews and a survey to understand which interventions are accepted across academic disciplines and approved by misinformation experts. Four interventions are supported by more than two in three misinformation experts: promoting media literacy, education in schools and universities, finding information about claims, and finding sources for claims. The most controversial intervention is deleting misinformation. We discuss the potentials and risks of all interventions. Education-based interventions are perceived as the most helpful by misinformation experts. Interventions focused on providing evidence are also widely perceived as helpful. We discuss them as scalable and always available interventions that empower users to independently identify misinformation. We also introduce the Phase Model of Misinformation Interventions that helps practitioners make informed decisions about which interventions to focus on and how to best combine interventions.
△ Less
Submitted 14 January, 2025;
originally announced January 2025.
-
Design Frictions on Social Media: Balancing Reduced Mindless Scrolling and User Satisfaction
Authors:
Nicolas Ruiz,
Gabriela Molina León,
Hendrik Heuer
Abstract:
Design features of social media platforms, such as infinite scroll, increase users' likelihood of experiencing normative dissociation -- a mental state of absorption that diminishes self-awareness and disrupts memory. This paper investigates how adding design frictions into the interface of a social media platform reduce mindless scrolling and user satisfaction. We conducted a study with 30 partic…
▽ More
Design features of social media platforms, such as infinite scroll, increase users' likelihood of experiencing normative dissociation -- a mental state of absorption that diminishes self-awareness and disrupts memory. This paper investigates how adding design frictions into the interface of a social media platform reduce mindless scrolling and user satisfaction. We conducted a study with 30 participants and compared their memory recognition of posts in two scenarios: one where participants had to react to each post to access further content and another using an infinite scroll design. Participants who used the design frictions interface exhibited significantly better content recall, although a majority of participants found the interface frustrating. We discuss design recommendations and scenarios where adding design frictions to social media platforms can be beneficial.
△ Less
Submitted 27 November, 2024; v1 submitted 26 July, 2024;
originally announced July 2024.
-
Reliability Criteria for News Websites
Authors:
Hendrik Heuer,
Elena Leah Glassman
Abstract:
Misinformation poses a threat to democracy and to people's health. Reliability criteria for news websites can help people identify misinformation. But despite their importance, there has been no empirically substantiated list of criteria for distinguishing reliable from unreliable news websites. We identify reliability criteria, describe how they are applied in practice, and compare them to prior…
▽ More
Misinformation poses a threat to democracy and to people's health. Reliability criteria for news websites can help people identify misinformation. But despite their importance, there has been no empirically substantiated list of criteria for distinguishing reliable from unreliable news websites. We identify reliability criteria, describe how they are applied in practice, and compare them to prior work. Based on our analysis, we distinguish between manipulable and less manipulable criteria and compare politically diverse laypeople as end users and journalists as expert users. We discuss 11 widely recognized criteria, including the following 6 criteria that are difficult to manipulate: content, political alignment, authors, professional standards, what sources are used, and a website's reputation. Finally, we describe how technology may be able to support people in applying these criteria in practice to assess the reliability of websites.
△ Less
Submitted 4 July, 2024;
originally announced July 2024.
-
Writer-Defined AI Personas for On-Demand Feedback Generation
Authors:
Karim Benharrak,
Tim Zindulka,
Florian Lehmann,
Hendrik Heuer,
Daniel Buschek
Abstract:
Compelling writing is tailored to its audience. This is challenging, as writers may struggle to empathize with readers, get feedback in time, or gain access to the target group. We propose a concept that generates on-demand feedback, based on writer-defined AI personas of any target audience. We explore this concept with a prototype (using GPT-3.5) in two user studies (N=5 and N=11): Writers appre…
▽ More
Compelling writing is tailored to its audience. This is challenging, as writers may struggle to empathize with readers, get feedback in time, or gain access to the target group. We propose a concept that generates on-demand feedback, based on writer-defined AI personas of any target audience. We explore this concept with a prototype (using GPT-3.5) in two user studies (N=5 and N=11): Writers appreciated the concept and strategically used personas for getting different perspectives. The feedback was seen as helpful and inspired revisions of text and personas, although it was often verbose and unspecific. We discuss the impact of on-demand feedback, the limited representativity of contemporary AI systems, and further ideas for defining AI personas. This work contributes to the vision of supporting writers with AI by expanding the socio-technical perspective in AI tool design: To empower creators, we also need to keep in mind their relationship to an audience.
△ Less
Submitted 20 February, 2024; v1 submitted 19 September, 2023;
originally announced September 2023.
-
ClimSim-Online: A Large Multi-scale Dataset and Framework for Hybrid ML-physics Climate Emulation
Authors:
Sungduk Yu,
Zeyuan Hu,
Akshay Subramaniam,
Walter Hannah,
Liran Peng,
Jerry Lin,
Mohamed Aziz Bhouri,
Ritwik Gupta,
Björn Lütjens,
Justus C. Will,
Gunnar Behrens,
Julius J. M. Busecke,
Nora Loose,
Charles I. Stern,
Tom Beucler,
Bryce Harrop,
Helge Heuer,
Benjamin R. Hillman,
Andrea Jenney,
Nana Liu,
Alistair White,
Tian Zheng,
Zhiming Kuang,
Fiaz Ahmed,
Elizabeth Barnes
, et al. (22 additional authors not shown)
Abstract:
Modern climate projections lack adequate spatial and temporal resolution due to computational constraints, leading to inaccuracies in representing critical processes like thunderstorms that occur on the sub-resolution scale. Hybrid methods combining physics with machine learning (ML) offer faster, higher fidelity climate simulations by outsourcing compute-hungry, high-resolution simulations to ML…
▽ More
Modern climate projections lack adequate spatial and temporal resolution due to computational constraints, leading to inaccuracies in representing critical processes like thunderstorms that occur on the sub-resolution scale. Hybrid methods combining physics with machine learning (ML) offer faster, higher fidelity climate simulations by outsourcing compute-hungry, high-resolution simulations to ML emulators. However, these hybrid ML-physics simulations require domain-specific data and workflows that have been inaccessible to many ML experts. As an extension of the ClimSim dataset (Yu et al., 2024), we present ClimSim-Online, which also includes an end-to-end workflow for developing hybrid ML-physics simulators. The ClimSim dataset includes 5.7 billion pairs of multivariate input/output vectors, capturing the influence of high-resolution, high-fidelity physics on a host climate simulator's macro-scale state. The dataset is global and spans ten years at a high sampling frequency. We provide a cross-platform, containerized pipeline to integrate ML models into operational climate simulators for hybrid testing. We also implement various ML baselines, alongside a hybrid baseline simulator, to highlight the ML challenges of building stable, skillful emulators. The data (https://huggingface.co/datasets/LEAP/ClimSim_high-res) and code (https://leap-stc.github.io/ClimSim and https://github.com/leap-stc/climsim-online) are publicly released to support the development of hybrid ML-physics and high-fidelity climate simulations.
△ Less
Submitted 8 July, 2024; v1 submitted 14 June, 2023;
originally announced June 2023.
-
A Comparative Evaluation of Interventions Against Misinformation: Augmenting the WHO Checklist
Authors:
Hendrik Heuer,
Elena Leah Glassman
Abstract:
During the COVID-19 pandemic, the World Health Organization provided a checklist to help people distinguish between accurate and misinformation. In controlled experiments in the United States and Germany, we investigated the utility of this ordered checklist and designed an interactive version to lower the cost of acting on checklist items. Across interventions, we observe non-trivial differences…
▽ More
During the COVID-19 pandemic, the World Health Organization provided a checklist to help people distinguish between accurate and misinformation. In controlled experiments in the United States and Germany, we investigated the utility of this ordered checklist and designed an interactive version to lower the cost of acting on checklist items. Across interventions, we observe non-trivial differences in participants' performance in distinguishing accurate and misinformation between the two countries and discuss some possible reasons that may predict the future helpfulness of the checklist in different environments. The checklist item that provides source labels was most frequently followed and was considered most helpful. Based on our empirical findings, we recommend practitioners focus on providing source labels rather than interventions that support readers performing their own fact-checks, even though this recommendation may be influenced by the WHO's chosen order. We discuss the complexity of providing such source labels and provide design recommendations.
△ Less
Submitted 13 December, 2022;
originally announced December 2022.
-
Beyond Participation: A Review of Co-Creation in Computing
Authors:
Juliane Jarke,
Gabriela Molina León,
Irina Zakharova,
Hendrik Heuer,
Ulrike Gerhard
Abstract:
New methods and technologies for engaging future users and other stakeholders in participatory (design) processes are being developed and proposed. Increasingly, researchers refer to co-creation in order to capture such approaches. However, how co-creation is being framed and understood across domains differs substantially. To better understand co-creation in computing, we conducted a literature r…
▽ More
New methods and technologies for engaging future users and other stakeholders in participatory (design) processes are being developed and proposed. Increasingly, researchers refer to co-creation in order to capture such approaches. However, how co-creation is being framed and understood across domains differs substantially. To better understand co-creation in computing, we conducted a literature review of all papers in the ACM Digital Library with co-creation or co-create in their abstracts. After an initial screening, we retained 62 for further analysis. We introduce a framework to analyze different notions of co-creation, distinguishing between co-creation target audiences, the roles of co-creators, the role of technology (as means or objective) and its results. We discuss the adoption of co-creation in domains such as learning, business, arts & culture, health, and the public sector. This paper contributes to the understanding of different approaches and conceptualizations of co-creation in computing and puts forward an agenda for future research.
△ Less
Submitted 8 November, 2021;
originally announced November 2021.
-
The Explanatory Gap in Algorithmic News Curation
Authors:
Hendrik Heuer
Abstract:
Considering the large amount of available content, social media platforms increasingly employ machine learning (ML) systems to curate news. This paper examines how well different explanations help expert users understand why certain news stories are recommended to them. The expert users were journalists, who are trained to judge the relevance of news. Surprisingly, none of the explanations are per…
▽ More
Considering the large amount of available content, social media platforms increasingly employ machine learning (ML) systems to curate news. This paper examines how well different explanations help expert users understand why certain news stories are recommended to them. The expert users were journalists, who are trained to judge the relevance of news. Surprisingly, none of the explanations are perceived as helpful. Our investigation provides a first indication of a gap between what is available to explain ML-based curation systems and what users need to understand such systems. We call this the Explanatory Gap in Machine Learning-based Curation Systems.
△ Less
Submitted 30 September, 2021;
originally announced September 2021.
-
Auditing the Biases Enacted by YouTube for Political Topics in Germany
Authors:
Hendrik Heuer,
Hendrik Hoch,
Andreas Breiter,
Yannis Theocharis
Abstract:
With YouTube's growing importance as a news platform, its recommendation system came under increased scrutiny. Recognizing YouTube's recommendation system as a broadcaster of media, we explore the applicability of laws that require broadcasters to give important political, ideological, and social groups adequate opportunity to express themselves in the broadcasted program of the service. We presen…
▽ More
With YouTube's growing importance as a news platform, its recommendation system came under increased scrutiny. Recognizing YouTube's recommendation system as a broadcaster of media, we explore the applicability of laws that require broadcasters to give important political, ideological, and social groups adequate opportunity to express themselves in the broadcasted program of the service. We present audits as an important tool to enforce such laws and to ensure that a system operates in the public's interest. To examine whether YouTube is enacting certain biases, we collected video recommendations about political topics by following chains of ten recommendations per video. Our findings suggest that YouTube's recommendation system is enacting important biases. We find that YouTube is recommending increasingly popular but topically unrelated videos. The sadness evoked by the recommended videos decreases while the happiness increases. We discuss the strong popularity bias we identified and analyze the link between the popularity of content and emotions. We also discuss how audits empower researchers and civic hackers to monitor complex machine learning (ML)-based systems like YouTube's recommendation system.
△ Less
Submitted 21 July, 2021;
originally announced July 2021.
-
Audit, Don't Explain -- Recommendations Based on a Socio-Technical Understanding of ML-Based Systems
Authors:
Hendrik Heuer
Abstract:
In this position paper, I provide a socio-technical perspective on machine learning-based systems. I also explain why systematic audits may be preferable to explainable AI systems. I make concrete recommendations for how institutions governed by public law akin to the German TÜV and Stiftung Warentest can ensure that ML systems operate in the interest of the public.
In this position paper, I provide a socio-technical perspective on machine learning-based systems. I also explain why systematic audits may be preferable to explainable AI systems. I make concrete recommendations for how institutions governed by public law akin to the German TÜV and Stiftung Warentest can ensure that ML systems operate in the interest of the public.
△ Less
Submitted 21 July, 2021;
originally announced July 2021.
-
Helping People Deal With Disinformation -- A Socio-Technical Perspective
Authors:
Hendrik Heuer
Abstract:
At the latest since the advent of the Internet, disinformation and conspiracy theories have become ubiquitous. Recent examples like QAnon and Pizzagate prove that false information can lead to real violence. In this motivation statement for the Workshop on Human Aspects of Misinformation at CHI 2021, I explain my research agenda focused on 1. why people believe in disinformation, 2. how people can…
▽ More
At the latest since the advent of the Internet, disinformation and conspiracy theories have become ubiquitous. Recent examples like QAnon and Pizzagate prove that false information can lead to real violence. In this motivation statement for the Workshop on Human Aspects of Misinformation at CHI 2021, I explain my research agenda focused on 1. why people believe in disinformation, 2. how people can be best supported in recognizing disinformation, and 3. what the potentials and risks of different tools designed to fight disinformation are.
△ Less
Submitted 9 April, 2021;
originally announced April 2021.
-
Methods for the Design and Evaluation of HCI+NLP Systems
Authors:
Hendrik Heuer,
Daniel Buschek
Abstract:
HCI and NLP traditionally focus on different evaluation methods. While HCI involves a small number of people directly and deeply, NLP traditionally relies on standardized benchmark evaluations that involve a larger number of people indirectly. We present five methodological proposals at the intersection of HCI and NLP and situate them in the context of ML-based NLP models. Our goal is to foster in…
▽ More
HCI and NLP traditionally focus on different evaluation methods. While HCI involves a small number of people directly and deeply, NLP traditionally relies on standardized benchmark evaluations that involve a larger number of people indirectly. We present five methodological proposals at the intersection of HCI and NLP and situate them in the context of ML-based NLP models. Our goal is to foster interdisciplinary collaboration and progress in both fields by emphasizing what the fields can learn from each other.
△ Less
Submitted 26 February, 2021;
originally announced February 2021.
-
Middle-Aged Video Consumers' Beliefs About Algorithmic Recommendations on YouTube
Authors:
Oscar Alvarado,
Hendrik Heuer,
Vero Vanden Abeele,
Andreas Breiter,
Katrien Verbert
Abstract:
User beliefs about algorithmic systems are constantly co-produced through user interaction and the complex socio-technical systems that generate recommendations. Identifying these beliefs is crucial because they influence how users interact with recommendation algorithms. With no prior work on user beliefs of algorithmic video recommendations, practitioners lack relevant knowledge to improve the u…
▽ More
User beliefs about algorithmic systems are constantly co-produced through user interaction and the complex socio-technical systems that generate recommendations. Identifying these beliefs is crucial because they influence how users interact with recommendation algorithms. With no prior work on user beliefs of algorithmic video recommendations, practitioners lack relevant knowledge to improve the user experience of such systems. To address this problem, we conducted semi-structured interviews with middle-aged YouTube video consumers to analyze their user beliefs about the video recommendation system. Our analysis revealed different factors that users believe influence their recommendations. Based on these factors, we identified four groups of user beliefs: Previous Actions, Social Media, Recommender System, and Company Policy. Additionally, we propose a framework to distinguish the four main actors that users believe influence their video recommendations: the current user, other users, the algorithm, and the organization. This framework provides a new lens to explore design suggestions based on the agency of these four actors. It also exposes a novel aspect previously unexplored: the effect of corporate decisions on the interaction with algorithmic recommendations. While we found that users are aware of the existence of the recommendation system on YouTube, we show that their understanding of this system is limited.
△ Less
Submitted 7 August, 2020;
originally announced August 2020.
-
How Fake News Affect Trust in the Output of a Machine Learning System for News Curation
Authors:
Hendrik Heuer,
Andreas Breiter
Abstract:
People are increasingly consuming news curated by machine learning (ML) systems. Motivated by studies on algorithmic bias, this paper explores which recommendations of an algorithmic news curation system users trust and how this trust is affected by untrustworthy news stories like fake news. In a study with 82 vocational school students with a background in IT, we found that users are able to prov…
▽ More
People are increasingly consuming news curated by machine learning (ML) systems. Motivated by studies on algorithmic bias, this paper explores which recommendations of an algorithmic news curation system users trust and how this trust is affected by untrustworthy news stories like fake news. In a study with 82 vocational school students with a background in IT, we found that users are able to provide trust ratings that distinguish trustworthy recommendations of quality news stories from untrustworthy recommendations. However, a single untrustworthy news story combined with four trustworthy news stories is rated similarly as five trustworthy news stories. The results could be a first indication that untrustworthy news stories benefit from appearing in a trustworthy context. The results also show the limitations of users' abilities to rate the recommendations of a news curation system. We discuss the implications of this for the user experience of interactive machine learning systems.
△ Less
Submitted 5 August, 2020;
originally announced August 2020.
-
More Than Accuracy: Towards Trustworthy Machine Learning Interfaces for Object Recognition
Authors:
Hendrik Heuer,
Andreas Breiter
Abstract:
This paper investigates the user experience of visualizations of a machine learning (ML) system that recognizes objects in images. This is important since even good systems can fail in unexpected ways as misclassifications on photo-sharing websites showed. In our study, we exposed users with a background in ML to three visualizations of three systems with different levels of accuracy. In interview…
▽ More
This paper investigates the user experience of visualizations of a machine learning (ML) system that recognizes objects in images. This is important since even good systems can fail in unexpected ways as misclassifications on photo-sharing websites showed. In our study, we exposed users with a background in ML to three visualizations of three systems with different levels of accuracy. In interviews, we explored how the visualization helped users assess the accuracy of systems in use and how the visualization and the accuracy of the system affected trust and reliance. We found that participants do not only focus on accuracy when assessing ML systems. They also take the perceived plausibility and severity of misclassification into account and prefer seeing the probability of predictions. Semantically plausible errors are judged as less severe than errors that are implausible, which means that system accuracy could be communicated through the types of errors.
△ Less
Submitted 5 August, 2020;
originally announced August 2020.
-
Is It Worth the Attention? A Comparative Evaluation of Attention Layers for Argument Unit Segmentation
Authors:
Maximilian Spliethöver,
Jonas Klaff,
Hendrik Heuer
Abstract:
Attention mechanisms have seen some success for natural language processing downstream tasks in recent years and generated new State-of-the-Art results. A thorough evaluation of the attention mechanism for the task of Argumentation Mining is missing, though. With this paper, we report a comparative evaluation of attention layers in combination with a bidirectional long short-term memory network, w…
▽ More
Attention mechanisms have seen some success for natural language processing downstream tasks in recent years and generated new State-of-the-Art results. A thorough evaluation of the attention mechanism for the task of Argumentation Mining is missing, though. With this paper, we report a comparative evaluation of attention layers in combination with a bidirectional long short-term memory network, which is the current state-of-the-art approach to the unit segmentation task. We also compare sentence-level contextualized word embeddings to pre-generated ones. Our findings suggest that for this task the additional attention layer does not improve upon a less complex approach. In most cases, the contextualized embeddings do also not show an improvement on the baseline score.
△ Less
Submitted 24 June, 2019;
originally announced June 2019.
-
Generating captions without looking beyond objects
Authors:
Hendrik Heuer,
Christof Monz,
Arnold W. M. Smeulders
Abstract:
This paper explores new evaluation perspectives for image captioning and introduces a noun translation task that achieves comparative image caption generation performance by translating from a set of nouns to captions. This implies that in image captioning, all word categories other than nouns can be evoked by a powerful language model without sacrificing performance on n-gram precision. The paper…
▽ More
This paper explores new evaluation perspectives for image captioning and introduces a noun translation task that achieves comparative image caption generation performance by translating from a set of nouns to captions. This implies that in image captioning, all word categories other than nouns can be evoked by a powerful language model without sacrificing performance on n-gram precision. The paper also investigates lower and upper bounds of how much individual word categories in the captions contribute to the final BLEU score. A large possible improvement exists for nouns, verbs, and prepositions.
△ Less
Submitted 18 October, 2016; v1 submitted 12 October, 2016;
originally announced October 2016.
-
Text comparison using word vector representations and dimensionality reduction
Authors:
Hendrik Heuer
Abstract:
This paper describes a technique to compare large text sources using word vector representations (word2vec) and dimensionality reduction (t-SNE) and how it can be implemented using Python. The technique provides a bird's-eye view of text sources, e.g. text summaries and their source material, and enables users to explore text sources like a geographical map. Word vector representations capture man…
▽ More
This paper describes a technique to compare large text sources using word vector representations (word2vec) and dimensionality reduction (t-SNE) and how it can be implemented using Python. The technique provides a bird's-eye view of text sources, e.g. text summaries and their source material, and enables users to explore text sources like a geographical map. Word vector representations capture many linguistic properties such as gender, tense, plurality and even semantic concepts like "capital city of". Using dimensionality reduction, a 2D map can be computed where semantically similar words are close to each other. The technique uses the word2vec model from the gensim Python library and t-SNE from scikit-learn.
△ Less
Submitted 2 July, 2016;
originally announced July 2016.
