Showing 1–9 of 9 results for author: Manzoor, E

Designing Effective Music Excerpts

Authors: Emaad Manzoor, Nikhil Malik

Abstract: Excerpts are widely used to preview and promote musical works. Effective excerpts induce consumption of the source musical work and thus generate revenue. Yet, what makes an excerpt effective remains unexplored. We leverage a policy change by Apple that generates quasi-exogenous variation in the excerpts of songs in the iTunes Music Store to estimate that having a 60 second longer excerpt increase… ▽ More Excerpts are widely used to preview and promote musical works. Effective excerpts induce consumption of the source musical work and thus generate revenue. Yet, what makes an excerpt effective remains unexplored. We leverage a policy change by Apple that generates quasi-exogenous variation in the excerpts of songs in the iTunes Music Store to estimate that having a 60 second longer excerpt increases songs' unique monthly listeners by 5.4% on average, by 9.7% for lesser known songs, and by 11.1% for lesser known artists. This is comparable to the impact of being featured on the Spotify Global Top 50 playlist. We develop measures of musical repetition and unpredictability to examine information provision as a mechanism, and find that the demand-enhancing effect of longer excerpts is suppressed when they are repetitive, too predictable, or too unpredictable. Our findings support platforms' adoption of longer excerpts to improve content discovery and our measures can help inform excerpt selection in practice. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Does Machine Learning Amplify Pricing Errors in the Housing Market? -- The Economics of Machine Learning Feedback Loops

Authors: Nikhil Malik, Emaad Manzoor

Abstract: Machine learning algorithms are increasingly employed to price or value homes for sale, properties for rent, rides for hire, and various other goods and services. Machine learning-based prices are typically generated by complex algorithms trained on historical sales data. However, displaying these prices to consumers anchors the realized sales prices, which will in turn become training samples for… ▽ More Machine learning algorithms are increasingly employed to price or value homes for sale, properties for rent, rides for hire, and various other goods and services. Machine learning-based prices are typically generated by complex algorithms trained on historical sales data. However, displaying these prices to consumers anchors the realized sales prices, which will in turn become training samples for future iterations of the algorithms. The economic implications of this machine learning "feedback loop" - an indirect human-algorithm interaction - remain relatively unexplored. In this work, we develop an analytical model of machine learning feedback loops in the context of the housing market. We show that feedback loops lead machine learning algorithms to become overconfident in their own accuracy (by underestimating its error), and leads home sellers to over-rely on possibly erroneous algorithmic prices. As a consequence at the feedback loop equilibrium, sale prices can become entirely erratic (relative to true consumer preferences in absence of ML price interference). We then identify conditions (choice of ML models, seller characteristics and market characteristics) where the economic payoffs for home sellers at the feedback loop equilibrium is worse off than no machine learning. We also empirically validate primitive building blocks of our analytical model using housing market data from Zillow. We conclude by prescribing algorithmic corrective strategies to mitigate the effects of machine learning feedback loops, discuss the incentives for platforms to adopt these strategies, and discuss the role of policymakers in regulating the same. △ Less

Submitted 18 February, 2023; originally announced February 2023.

Expanding Knowledge Graphs with Humans in the Loop

Authors: Emaad Manzoor, Jordan Tong, Sriniketh Vijayaraghavan, Rui Li

Abstract: Curated knowledge graphs encode domain expertise and improve the performance of recommendation, segmentation, ad targeting, and other machine learning systems in several domains. As new concepts emerge in a domain, knowledge graphs must be expanded to preserve machine learning performance. Manually expanding knowledge graphs, however, is infeasible at scale. In this work, we propose a method for k… ▽ More Curated knowledge graphs encode domain expertise and improve the performance of recommendation, segmentation, ad targeting, and other machine learning systems in several domains. As new concepts emerge in a domain, knowledge graphs must be expanded to preserve machine learning performance. Manually expanding knowledge graphs, however, is infeasible at scale. In this work, we propose a method for knowledge graph expansion with humans-in-the-loop. Concretely, given a knowledge graph, our method predicts the "parents" of new concepts to be added to this graph for further verification by human experts. We show that our method is both accurate and provably "human-friendly". Specifically, we prove that our method predicts parents that are "near" concepts' true parents in the knowledge graph, even when the predictions are incorrect. We then show, with a controlled experiment, that satisfying this property increases both the speed and the accuracy of the human-algorithm collaboration. We further evaluate our method on a knowledge graph from Pinterest and show that it outperforms competing methods on both accuracy and human-friendliness. Upon deployment in production at Pinterest, our method reduced the time needed for knowledge graph expansion by ~400% (compared to manual expansion), and contributed to a subsequent increase in ad revenue of 20%. △ Less

Submitted 26 March, 2023; v1 submitted 9 December, 2022; originally announced December 2022.

Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond

Authors: Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, Diyi Yang

Abstract: A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and social sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinction is beginning to fade, with an emerging area of interdisciplinary research at the conver… ▽ More A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and social sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinction is beginning to fade, with an emerging area of interdisciplinary research at the convergence of causal inference and language processing. Still, research on causality in NLP remains scattered across domains without unified definitions, benchmark datasets and clear articulations of the challenges and opportunities in the application of causal inference to the textual domain, with its unique properties. In this survey, we consolidate research across academic areas and situate it in the broader NLP landscape. We introduce the statistical challenge of estimating causal effects with text, encompassing settings where text is used as an outcome, treatment, or to address confounding. In addition, we explore potential uses of causal inference to improve the robustness, fairness, and interpretability of NLP models. We thus provide a unified overview of causal inference for the NLP community. △ Less

Submitted 30 July, 2022; v1 submitted 2 September, 2021; originally announced September 2021.

Comments: Accepted to Transactions of the Association for Computational Linguistics (TACL)

Uncovering Latent Biases in Text: Method and Application to Peer Review

Authors: Emaad Manzoor, Nihar B. Shah

Abstract: Quantifying systematic disparities in numerical quantities such as employment rates and wages between population subgroups provides compelling evidence for the existence of societal biases. However, biases in the text written for members of different subgroups (such as in recommendation letters for male and non-male candidates), though widely reported anecdotally, remain challenging to quantify. I… ▽ More Quantifying systematic disparities in numerical quantities such as employment rates and wages between population subgroups provides compelling evidence for the existence of societal biases. However, biases in the text written for members of different subgroups (such as in recommendation letters for male and non-male candidates), though widely reported anecdotally, remain challenging to quantify. In this work, we introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators. We develop a nonparametric estimation and inference procedure to estimate this bias. We then formalize an identification strategy to causally link the estimated bias to the visibility of subgroup membership indicators, provided observations from time periods both before and after an identity-hiding policy change. We identify an application wherein "ground truth" bias can be inferred to evaluate our framework, instead of relying on synthetic or secondary data. Specifically, we apply our framework to quantify biases in the text of peer reviews from a reputed machine learning conference before and after the conference adopted a double-blind reviewing policy. We show evidence of biases in the review ratings that serves as "ground truth", and show that our proposed framework accurately detects these biases from the review text without having access to the review ratings. △ Less

Submitted 28 October, 2020; originally announced October 2020.

Detecting Attackable Sentences in Arguments

Authors: Yohan Jo, Seojin Bang, Emaad Manzoor, Eduard Hovy, Chris Reed

Abstract: Finding attackable sentences in an argument is the first step toward successful refutation in argumentation. We present a first large-scale analysis of sentence attackability in online arguments. We analyze driving reasons for attacks in argumentation and identify relevant characteristics of sentences. We demonstrate that a sentence's attackability is associated with many of these characteristics… ▽ More Finding attackable sentences in an argument is the first step toward successful refutation in argumentation. We present a first large-scale analysis of sentence attackability in online arguments. We analyze driving reasons for attacks in argumentation and identify relevant characteristics of sentences. We demonstrate that a sentence's attackability is associated with many of these characteristics regarding the sentence's content, proposition types, and tone, and that an external knowledge source can provide useful information about attackability. Building on these findings, we demonstrate that machine learning models can automatically detect attackable sentences in arguments, significantly better than several baselines and comparably well to laypeople. △ Less

Submitted 6 October, 2020; originally announced October 2020.

Comments: EMNLP 2020

Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online

Authors: Emaad Manzoor, George H. Chen, Dokyun Lee, Michael D. Smith

Abstract: Deliberation among individuals online plays a key role in shaping the opinions that drive votes, purchases, donations and other critical offline behavior. Yet, the determinants of opinion-change via persuasion in deliberation online remain largely unexplored. Our research examines the persuasive power of $\textit{ethos}$ -- an individual's "reputation" -- using a 7-year panel of over a million deb… ▽ More Deliberation among individuals online plays a key role in shaping the opinions that drive votes, purchases, donations and other critical offline behavior. Yet, the determinants of opinion-change via persuasion in deliberation online remain largely unexplored. Our research examines the persuasive power of $\textit{ethos}$ -- an individual's "reputation" -- using a 7-year panel of over a million debates from an argumentation platform containing explicit indicators of successful persuasion. We identify the causal effect of reputation on persuasion by constructing an instrument for reputation from a measure of past debate competition, and by controlling for unstructured argument text using neural models of language in the double machine-learning framework. We find that an individual's reputation significantly impacts their persuasion rate above and beyond the validity, strength and presentation of their arguments. In our setting, we find that having 10 additional reputation points causes a 31% increase in the probability of successful persuasion over the platform average. We also find that the impact of reputation is moderated by characteristics of the argument content, in a manner consistent with a theoretical model that attributes the persuasive power of reputation to heuristic information-processing under cognitive overload. We discuss managerial implications for platforms that facilitate deliberative decision-making for public and private organizations online. △ Less

Submitted 1 June, 2020; originally announced June 2020.

Scheduling Broadcasts in a Network of Timelines

Authors: Emaad Manzoor, Haewoon Kwak, Panos Kalnis

Abstract: Broadcasts and timelines are the primary mechanism of information exchange in online social platforms today. Services like Facebook, Twitter and Instagram have enabled ordinary people to reach large audiences spanning cultures and countries, while their massive popularity has created increasingly competitive marketplaces of attention. Timing broadcasts to capture the attention of such geographical… ▽ More Broadcasts and timelines are the primary mechanism of information exchange in online social platforms today. Services like Facebook, Twitter and Instagram have enabled ordinary people to reach large audiences spanning cultures and countries, while their massive popularity has created increasingly competitive marketplaces of attention. Timing broadcasts to capture the attention of such geographically diverse audiences has sparked interest from many startups and social marketing gurus. However, formal study is lacking on both the timing and frequency problems. We study for the first time the broadcast scheduling problem of specifying the timing and frequency of publishing content to maximise the attention received. We validate and quantify three interacting behavioural phenomena to parametrise social platform users: information overload, bursty circadian rhythms and monotony aversion, which is defined here for the first time. We formalise a timeline information exchange process based on these phenomena, and formulate an objective function that quantifies the expected collective attention. We finally present experiments on real data from Twitter, where we discover a counter-intuitive scheduling strategy that outperforms popular heuristics while producing fewer posts. △ Less

Submitted 19 October, 2016; originally announced October 2016.

Comments: Submitted to KDD 2015

Fast Memory-efficient Anomaly Detection in Streaming Heterogeneous Graphs

Authors: Emaad A. Manzoor, Sadegh Momeni, Venkat N. Venkatakrishnan, Leman Akoglu

Abstract: Given a stream of heterogeneous graphs containing different types of nodes and edges, how can we spot anomalous ones in real-time while consuming bounded memory? This problem is motivated by and generalizes from its application in security to host-level advanced persistent threat (APT) detection. We propose StreamSpot, a clustering based anomaly detection approach that addresses challenges in two… ▽ More Given a stream of heterogeneous graphs containing different types of nodes and edges, how can we spot anomalous ones in real-time while consuming bounded memory? This problem is motivated by and generalizes from its application in security to host-level advanced persistent threat (APT) detection. We propose StreamSpot, a clustering based anomaly detection approach that addresses challenges in two key fronts: (1) heterogeneity, and (2) streaming nature. We introduce a new similarity function for heterogeneous graphs that compares two graphs based on their relative frequency of local substructures, represented as short strings. This function lends itself to a vector representation of a graph, which is (a) fast to compute, and (b) amenable to a sketched version with bounded size that preserves similarity. StreamSpot exhibits desirable properties that a streaming application requires---it is (i) fully-streaming; processing the stream one edge at a time as it arrives, (ii) memory-efficient; requiring constant space for the sketches and the clustering, (iii) fast; taking constant time to update the graph sketches and the cluster summaries that can process over 100K edges per second, and (iv) online; scoring and flagging anomalies in real time. Experiments on datasets containing simulated system-call flow graphs from normal browser activity and various attack scenarios (ground truth) show that our proposed StreamSpot is high-performance; achieving above 95% detection accuracy with small delay, as well as competitive time and memory usage. △ Less

Submitted 22 February, 2016; v1 submitted 15 February, 2016; originally announced February 2016.

Comments: 10 pages, 2 tables, 14 figures