Showing 1–17 of 17 results for author: Harris, K D

Spectral gap-based deterministic tensor completion

Authors: Kameron Decker Harris, Oscar López, Angus Read, Yizhe Zhu

Abstract: Tensor completion is a core machine learning algorithm used in recommender systems and other domains with missing data. While the matrix case is well-understood, theoretical results for tensor problems are limited, particularly when the sampling patterns are deterministic. Here we bound the generalization error of the solutions of two tensor completion methods, Poisson loss and atomic norm minimiz… ▽ More Tensor completion is a core machine learning algorithm used in recommender systems and other domains with missing data. While the matrix case is well-understood, theoretical results for tensor problems are limited, particularly when the sampling patterns are deterministic. Here we bound the generalization error of the solutions of two tensor completion methods, Poisson loss and atomic norm minimization, providing tighter bounds in terms of the target tensor rank. If the ground-truth tensor is order $t$ with CP-rank $r$, the dependence on $r$ is improved from $r^{2(t-1)(t^2-t-1)}$ in arXiv:1910.10692 to $r^{2(t-1)(3t-5)}$. The error in our bounds is deterministically controlled by the spectral gap of the sampling sparsity pattern. We also prove several new properties for the atomic tensor norm, reducing the rank dependence from $r^{3t-3}$ in arXiv:1711.04965 to $r^{3t-5}$ under random sampling schemes. A limitation is that atomic norm minimization, while theoretically interesting, leads to inefficient algorithms. However, numerical experiments illustrate the dependence of the reconstruction error on the spectral gap for the practical max-quasinorm, ridge penalty, and Poisson loss minimization algorithms. This view through the spectral gap is a promising window for further study of tensor algorithms. △ Less

Submitted 9 June, 2023; originally announced June 2023.

Comments: In proceedings of SampTA 2023

On 1/n neural representation and robustness

Authors: Josue Nassar, Piotr Aleksander Sokol, SueYeon Chung, Kenneth D. Harris, Il Memming Park

Abstract: Understanding the nature of representation in neural networks is a goal shared by neuroscience and machine learning. It is therefore exciting that both fields converge not only on shared questions but also on similar approaches. A pressing question in these areas is understanding how the structure of the representation used by neural networks affects both their generalization, and robustness to pe… ▽ More Understanding the nature of representation in neural networks is a goal shared by neuroscience and machine learning. It is therefore exciting that both fields converge not only on shared questions but also on similar approaches. A pressing question in these areas is understanding how the structure of the representation used by neural networks affects both their generalization, and robustness to perturbations. In this work, we investigate the latter by juxtaposing experimental results regarding the covariance spectrum of neural representations in the mouse V1 (Stringer et al) with artificial neural networks. We use adversarial robustness to probe Stringer et al's theory regarding the causal role of a 1/n covariance spectrum. We empirically investigate the benefits such a neural code confers in neural networks, and illuminate its role in multi-layer architectures. Our results show that imposing the experimentally observed structure on artificial neural networks makes them more robust to adversarial attacks. Moreover, our findings complement the existing theory relating wide neural networks to kernel methods, by showing the role of intermediate representations. △ Less

Submitted 8 December, 2020; originally announced December 2020.

Deterministic tensor completion with hypergraph expanders

Authors: Kameron Decker Harris, Yizhe Zhu

Abstract: We provide a novel analysis of low-rank tensor completion based on hypergraph expanders. As a proxy for rank, we minimize the max-quasinorm of the tensor, which generalizes the max-norm for matrices. Our analysis is deterministic and shows that the number of samples required to approximately recover an order-$t$ tensor with at most $n$ entries per dimension is linear in $n$, under the assumption t… ▽ More We provide a novel analysis of low-rank tensor completion based on hypergraph expanders. As a proxy for rank, we minimize the max-quasinorm of the tensor, which generalizes the max-norm for matrices. Our analysis is deterministic and shows that the number of samples required to approximately recover an order-$t$ tensor with at most $n$ entries per dimension is linear in $n$, under the assumption that the rank and order of the tensor are $O(1)$. As steps in our proof, we find a new expander mixing lemma for a $t$-partite, $t$-uniform regular hypergraph model, and prove several new properties about tensor max-quasinorm. To the best of our knowledge, this is the first deterministic analysis of tensor completion. We develop a practical algorithm that solves a relaxed version of the max-quasinorm minimization problem, and we demonstrate its efficacy with numerical experiments. △ Less

Submitted 29 July, 2021; v1 submitted 23 October, 2019; originally announced October 2019.

Comments: 35 pages, 4 figures. To appear in SIAM Journal on Mathematics of Data Science

Journal ref: SIAM Journal on Mathematics of Data Science, 3(4), 1117-1140, 2021

Additive function approximation in the brain

Authors: Kameron Decker Harris

Abstract: Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned. A network with $d$ inputs per neuron is found to be equivalent to an additive model of order… ▽ More Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned. A network with $d$ inputs per neuron is found to be equivalent to an additive model of order $d$, whereas with a degree distribution the network combines additive terms of different orders. We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable. Thus, even simple brain architectures can be powerful function approximators. Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists. △ Less

Submitted 13 September, 2019; v1 submitted 5 September, 2019; originally announced September 2019.

Journal ref: NeurIPS Real Neurons Hidden Units (NeuroAI workshop) 2019

Time-varying Autoregression with Low Rank Tensors

Authors: Kameron Decker Harris, Aleksandr Aravkin, Rajesh Rao, Bingni Wen Brunton

Abstract: We present a windowed technique to learn parsimonious time-varying autoregressive models from multivariate timeseries. This unsupervised method uncovers interpretable spatiotemporal structure in data via non-smooth and non-convex optimization. In each time window, we assume the data follow a linear model parameterized by a system matrix, and we model this stack of potentially different system matr… ▽ More We present a windowed technique to learn parsimonious time-varying autoregressive models from multivariate timeseries. This unsupervised method uncovers interpretable spatiotemporal structure in data via non-smooth and non-convex optimization. In each time window, we assume the data follow a linear model parameterized by a system matrix, and we model this stack of potentially different system matrices as a low rank tensor. Because of its structure, the model is scalable to high-dimensional data and can easily incorporate priors such as smoothness over time. We find the components of the tensor using alternating minimization and prove that any stationary point of this algorithm is a local minimum. We demonstrate on a synthetic example that our method identifies the true rank of a switching linear system in the presence of noise. We illustrate our model's utility and superior scalability over extant methods when applied to several synthetic and real-world example: two types of time-varying linear systems, worm behavior, sea surface temperature, and monkey brain datasets. △ Less

Submitted 19 May, 2020; v1 submitted 20 May, 2019; originally announced May 2019.

Characterizing the invariances of learning algorithms using category theory

Authors: Kenneth D. Harris

Abstract: Many learning algorithms have invariances: when their training data is transformed in certain ways, the function they learn transforms in a predictable manner. Here we formalize this notion using concepts from the mathematical field of category theory. The invariances that a supervised learning algorithm possesses are formalized by categories of predictor and target spaces, whose morphisms represe… ▽ More Many learning algorithms have invariances: when their training data is transformed in certain ways, the function they learn transforms in a predictable manner. Here we formalize this notion using concepts from the mathematical field of category theory. The invariances that a supervised learning algorithm possesses are formalized by categories of predictor and target spaces, whose morphisms represent the algorithm's invariances, and an index category whose morphisms represent permutations of the training examples. An invariant learning algorithm is a natural transformation between two functors from the product of these categories to the category of sets, representing training datasets and learned functions respectively. We illustrate the framework by characterizing and contrasting the invariances of linear regression and ridge regression. △ Less

Submitted 6 May, 2019; originally announced May 2019.

Reply to Garcia et al.: Common mistakes in measuring frequency dependent word characteristics

Authors: P. S. Dodds, E. M. Clark, S. Desu, M. R. Frank, A. J. Reagan, J. R. Williams, L. Mitchell, K. D. Harris, I. M. Kloumann, J. P. Bagrow, K. Megerdoomian, M. T. McMahon, B. F. Tivnan, C. M. Danforth

Abstract: We demonstrate that the concerns expressed by Garcia et al. are misplaced, due to (1) a misreading of our findings in [1]; (2) a widespread failure to examine and present words in support of asserted summary quantities based on word usage frequencies; and (3) a range of misconceptions about word usage frequency, word rank, and expert-constructed word lists. In particular, we show that the English… ▽ More We demonstrate that the concerns expressed by Garcia et al. are misplaced, due to (1) a misreading of our findings in [1]; (2) a widespread failure to examine and present words in support of asserted summary quantities based on word usage frequencies; and (3) a range of misconceptions about word usage frequency, word rank, and expert-constructed word lists. In particular, we show that the English component of our study compares well statistically with two related surveys, that no survey design influence is apparent, and that estimates of measurement error do not explain the positivity biases reported in our work and that of others. We further demonstrate that for the frequency dependence of positivity---of which we explored the nuances in great detail in [1]---Garcia et al. did not perform a reanalysis of our data---they instead carried out an analysis of a different, statistically improper data set and introduced a nonlinearity before performing linear regression. △ Less

Submitted 28 May, 2015; v1 submitted 25 May, 2015; originally announced May 2015.

Comments: 5 pages, 2 figures, 1 table. Expanded version of reply appearing in PNAS 2015

Human language reveals a universal positivity bias

Authors: Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, Karine Megerdoomian, Matthew T. McMahon, Brian F. Tivnan, Christopher M. Danforth

Abstract: Using human evaluation of 100,000 words spread across 24 corpora in 10 languages diverse in origin and culture, we present evidence of a deep imprint of human sociality in language, observing that (1) the words of natural human language possess a universal positivity bias; (2) the estimated emotional content of words is consistent between languages under translation; and (3) this positivity bias i… ▽ More Using human evaluation of 100,000 words spread across 24 corpora in 10 languages diverse in origin and culture, we present evidence of a deep imprint of human sociality in language, observing that (1) the words of natural human language possess a universal positivity bias; (2) the estimated emotional content of words is consistent between languages under translation; and (3) this positivity bias is strongly independent of frequency of word usage. Alongside these general regularities, we describe inter-language variations in the emotional spectrum of languages which allow us to rank corpora. We also show how our word evaluations can be used to construct physical-like instruments for both real-time and offline measurement of the emotional content of large-scale texts. △ Less

Submitted 15 June, 2014; originally announced June 2014.

Comments: Manuscript: 7 pages, 4 figures; Supplementary Material: 49 pages, 43 figures, 6 tables. Online appendices available at http://www.uvm.edu/storylab/share/papers/dodds2014a/

High-dimensional cluster analysis with the Masked EM Algorithm

Authors: Shabnam N. Kadir, Dan F. M. Goodman, Kenneth D. Harris

Abstract: Cluster analysis faces two problems in high dimensions: first, the `curse of dimensionality' that can lead to overfitting and poor generalization performance; and second, the sheer time taken for conventional algorithms to process large amounts of high-dimensional data. In many applications, only a small subset of features provide information about the cluster membership of any one data point, how… ▽ More Cluster analysis faces two problems in high dimensions: first, the `curse of dimensionality' that can lead to overfitting and poor generalization performance; and second, the sheer time taken for conventional algorithms to process large amounts of high-dimensional data. In many applications, only a small subset of features provide information about the cluster membership of any one data point, however this informative feature subset may not be the same for all data points. Here we introduce a `Masked EM' algorithm for fitting mixture of Gaussians models in such cases. We show that the algorithm performs close to optimally on simulated Gaussian data, and in an application of `spike sorting' of high channel-count neuronal recordings. △ Less

Submitted 11 September, 2013; originally announced September 2013.

Comments: 10 pages, 2 figures

Dynamical influence processes on networks: General theory and applications to social contagion

Authors: Kameron Decker Harris, Christopher M. Danforth, Peter Sheridan Dodds

Abstract: We study binary state dynamics on a network where each node acts in response to the average state of its neighborhood. Allowing varying amounts of stochasticity in both the network and node responses, we find different outcomes in random and deterministic versions of the model. In the limit of a large, dense network, however, we show that these dynamics coincide. We construct a general mean field… ▽ More We study binary state dynamics on a network where each node acts in response to the average state of its neighborhood. Allowing varying amounts of stochasticity in both the network and node responses, we find different outcomes in random and deterministic versions of the model. In the limit of a large, dense network, however, we show that these dynamics coincide. We construct a general mean field theory for random networks and show this predicts that the dynamics on the network are a smoothed version of the average response function dynamics. Thus, the behavior of the system can range from steady state to chaotic depending on the response functions, network connectivity, and update synchronicity. As a specific example, we model the competing tendencies of imitation and non-conformity by incorporating an off-threshold into standard threshold models of social contagion. In this way we attempt to capture important aspects of fashions and societal trends. We compare our theory to extensive simulations of this "limited imitation contagion" model on Poisson random graphs, finding agreement between the mean-field theory and stochastic simulations. △ Less

Submitted 7 July, 2014; v1 submitted 6 March, 2013; originally announced March 2013.

Comments: 12 pages, 7 figures

Journal ref: Physical Review E 88, 022816 (2013)

The Geography of Happiness: Connecting Twitter sentiment and expression, demographics, and objective characteristics of place

Authors: Lewis Mitchell, Kameron Decker Harris, Morgan R. Frank, Peter Sheridan Dodds, Christopher M. Danforth

Abstract: We conduct a detailed investigation of correlations between real-time expressions of individuals made across the United States and a wide range of emotional, geographic, demographic, and health characteristics. We do so by combining (1) a massive, geo-tagged data set comprising over 80 million words generated over the course of several recent years on the social network service Twitter and (2) ann… ▽ More We conduct a detailed investigation of correlations between real-time expressions of individuals made across the United States and a wide range of emotional, geographic, demographic, and health characteristics. We do so by combining (1) a massive, geo-tagged data set comprising over 80 million words generated over the course of several recent years on the social network service Twitter and (2) annually-surveyed characteristics of all 50 states and close to 400 urban populations. Among many results, we generate taxonomies of states and cities based on their similarities in word use; estimate the happiness levels of states and cities; correlate highly-resolved demographic characteristics with happiness levels; and connect word choice and message length with urban characteristics such as education levels and obesity rates. Our results show how social media may potentially be used to estimate real-time levels and changes in population-level measures such as obesity rates. △ Less

Submitted 18 May, 2013; v1 submitted 13 February, 2013; originally announced February 2013.

Journal ref: PLoS ONE 8(5): e64417, 2013

On-off Threshold Models of Social Contagion

Authors: Kameron Decker Harris

Abstract: We study binary state contagion dynamics on a social network where nodes act in response to the average state of their neighborhood. We model the competing tendencies of imitation and non-conformity by incorporating an off-threshold into standard threshold models of behavior. In this way, we attempt to capture important aspects of fashions and general societal trends. Allowing varying amounts of s… ▽ More We study binary state contagion dynamics on a social network where nodes act in response to the average state of their neighborhood. We model the competing tendencies of imitation and non-conformity by incorporating an off-threshold into standard threshold models of behavior. In this way, we attempt to capture important aspects of fashions and general societal trends. Allowing varying amounts of stochasticity in both the network and node responses, we find different outcomes in the random and deterministic versions of the model. In the limit of a large, dense network, however, we show that these dynamics coincide. The dynamical behavior of the system ranges from steady state to chaotic depending on network connectivity and update synchronicity. We construct a mean field theory for general random networks. In the undirected case, the mean field theory predicts that the dynamics on the network are a smoothed version of the average node response dynamics. We compare our theory to extensive simulations on Poisson random graphs with node responses that average to the chaotic tent map. △ Less

Submitted 10 September, 2012; originally announced September 2012.

Comments: Master's Thesis, mathematics, University of Vermont

Twitter reciprocal reply networks exhibit assortativity with respect to happiness

Authors: Catherine A. Bliss, Isabel M. Kloumann, Kameron Decker Harris, Christopher M. Danforth, Peter Sheridan Dodds

Abstract: The advent of social media has provided an extraordinary, if imperfect, 'big data' window into the form and evolution of social networks. Based on nearly 40 million message pairs posted to Twitter between September 2008 and February 2009, we construct and examine the revealed social network structure and dynamics over the time scales of days, weeks, and months. At the level of user behavior, we em… ▽ More The advent of social media has provided an extraordinary, if imperfect, 'big data' window into the form and evolution of social networks. Based on nearly 40 million message pairs posted to Twitter between September 2008 and February 2009, we construct and examine the revealed social network structure and dynamics over the time scales of days, weeks, and months. At the level of user behavior, we employ our recently developed hedonometric analysis methods to investigate patterns of sentiment expression. We find users' average happiness scores to be positively and significantly correlated with those of users one, two, and three links away. We strengthen our analysis by proposing and using a null model to test the effect of network topology on the assortativity of happiness. We also find evidence that more well connected users write happier status updates, with a transition occurring around Dunbar's number. More generally, our work provides evidence of a social sub-network structure within Twitter and raises several methodological points of interest with regard to social network reconstructions. △ Less

Submitted 11 May, 2012; v1 submitted 5 December, 2011; originally announced December 2011.

Comments: 22 pages, 21 figures, 5 tables, In press at the Journal of Computational Science

Positivity of the English language

Authors: Isabel M. Kloumann, Christopher M. Danforth, Kameron Decker Harris, Catherine A. Bliss, Peter Sheridan Dodds

Abstract: Over the last million years, human language has emerged and evolved as a fundamental instrument of social communication and semiotic representation. People use language in part to convey emotional information, leading to the central and contingent questions: (1) What is the emotional spectrum of natural language? and (2) Are natural languages neutrally, positively, or negatively biased? Here, we r… ▽ More Over the last million years, human language has emerged and evolved as a fundamental instrument of social communication and semiotic representation. People use language in part to convey emotional information, leading to the central and contingent questions: (1) What is the emotional spectrum of natural language? and (2) Are natural languages neutrally, positively, or negatively biased? Here, we report that the human-perceived positivity of over 10,000 of the most frequently used English words exhibits a clear positive bias. More deeply, we characterize and quantify distributions of word positivity for four large and distinct corpora, demonstrating that their form is broadly invariant with respect to frequency of word use. △ Less

Submitted 12 January, 2012; v1 submitted 25 August, 2011; originally announced August 2011.

Comments: Manuscript: 9 pages, 3 tables, 5 figures; Supplementary Information: 12 pages, 3 tables, 8 figures

Journal ref: PLoS ONE, Vol 7, e29484, 2012

Exact solutions for social and biological contagion models on mixed directed and undirected, degree-correlated random networks

Authors: Joshua L. Payne, Kameron Decker Harris, Peter Sheridan Dodds

Abstract: We derive analytic expressions for the possibility, probability, and expected size of global spreading events starting from a single infected seed for a broad collection of contagion processes acting on random networks with both directed and undirected edges and arbitrary degree-degree correlations. Our work extends previous theoretical developments for the undirected case, and we provide numerica… ▽ More We derive analytic expressions for the possibility, probability, and expected size of global spreading events starting from a single infected seed for a broad collection of contagion processes acting on random networks with both directed and undirected edges and arbitrary degree-degree correlations. Our work extends previous theoretical developments for the undirected case, and we provide numerical support for our findings by investigating an example class of networks for which we are able to obtain closed-form expressions. △ Less

Submitted 14 June, 2011; v1 submitted 28 February, 2011; originally announced March 2011.

Comments: 10 pages, 3 figures

Direct, physically-motivated derivation of the contagion condition for spreading processes on generalized random networks

Authors: Peter Sheridan Dodds, Kameron Decker Harris, Joshua L. Payne

Abstract: For a broad range single-seed contagion processes acting on generalized random networks, we derive a unifying analytic expression for the possibility of global spreading events in a straightforward, physically intuitive fashion. Our reasoning lays bare a direct mechanical understanding of an archetypal spreading phenomena that is not evident in circuitous extant mathematical approaches. For a broad range single-seed contagion processes acting on generalized random networks, we derive a unifying analytic expression for the possibility of global spreading events in a straightforward, physically intuitive fashion. Our reasoning lays bare a direct mechanical understanding of an archetypal spreading phenomena that is not evident in circuitous extant mathematical approaches. △ Less

Submitted 16 May, 2011; v1 submitted 28 January, 2011; originally announced January 2011.

Comments: 4 pages, 1 figure, 1 table

Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter

Authors: Peter Sheridan Dodds, Kameron Decker Harris, Isabel M. Kloumann, Catherine A. Bliss, Christopher M. Danforth

Abstract: Individual happiness is a fundamental societal metric. Normally measured through self-report, happiness has often been indirectly characterized and overshadowed by more readily quantifiable economic indicators such as gross domestic product. Here, we examine expressions made on the online, global microblog and social networking service Twitter, uncovering and explaining temporal variations in happ… ▽ More Individual happiness is a fundamental societal metric. Normally measured through self-report, happiness has often been indirectly characterized and overshadowed by more readily quantifiable economic indicators such as gross domestic product. Here, we examine expressions made on the online, global microblog and social networking service Twitter, uncovering and explaining temporal variations in happiness and information levels over timescales ranging from hours to years. Our data set comprises over 46 billion words contained in nearly 4.6 billion expressions posted over a 33 month span by over 63 million unique users. In measuring happiness, we use a real-time, remote-sensing, non-invasive, text-based approach---a kind of hedonometer. In building our metric, made available with this paper, we conducted a survey to obtain happiness evaluations of over 10,000 individual words, representing a tenfold size improvement over similar existing word sets. Rather than being ad hoc, our word list is chosen solely by frequency of usage and we show how a highly robust metric can be constructed and defended. △ Less

Submitted 8 December, 2011; v1 submitted 26 January, 2011; originally announced January 2011.

Comments: 27 pages, 17 figures, 3 tables. Supplementary Information: 1 table, 52 figures

Journal ref: PLoS ONE, Vol 6(2): e26752, 2011