Showing 1–10 of 10 results for author: McNamara, D

Globally Convergent Variational Inference

Authors: Declan McNamara, Jackson Loper, Jeffrey Regier

Abstract: In variational inference (VI), an approximation of the posterior distribution is selected from a family of distributions through numerical optimization. With the most common variational objective function, known as the evidence lower bound (ELBO), only convergence to a local optimum can be guaranteed. In this work, we instead establish the global convergence of a particular VI method. This VI meth… ▽ More In variational inference (VI), an approximation of the posterior distribution is selected from a family of distributions through numerical optimization. With the most common variational objective function, known as the evidence lower bound (ELBO), only convergence to a local optimum can be guaranteed. In this work, we instead establish the global convergence of a particular VI method. This VI method, which may be considered an instance of neural posterior estimation (NPE), minimizes an expectation of the inclusive (forward) KL divergence to fit a variational distribution that is parameterized by a neural network. Our convergence result relies on the neural tangent kernel (NTK) to characterize the gradient dynamics that arise from considering the variational objective in function space. In the asymptotic regime of a fixed, positive-definite neural tangent kernel, we establish conditions under which the variational objective admits a unique solution in a reproducing kernel Hilbert space (RKHS). Then, we show that the gradient descent dynamics in function space converge to this unique function. In ablation studies and practical problems, we demonstrate that our results explain the behavior of NPE in non-asymptotic finite-neuron settings, and show that NPE outperforms ELBO-based optimization, which often converges to shallow local optima. △ Less

Submitted 14 January, 2025; originally announced January 2025.

Comments: Accepted to the 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference

Authors: Declan McNamara, Jackson Loper, Jeffrey Regier

Abstract: For training an encoder network to perform amortized variational inference, the Kullback-Leibler (KL) divergence from the exact posterior to its approximation, known as the inclusive or forward KL, is an increasingly popular choice of variational objective due to the mass-covering property of its minimizer. However, minimizing this objective is challenging. A popular existing approach, Reweighted… ▽ More For training an encoder network to perform amortized variational inference, the Kullback-Leibler (KL) divergence from the exact posterior to its approximation, known as the inclusive or forward KL, is an increasingly popular choice of variational objective due to the mass-covering property of its minimizer. However, minimizing this objective is challenging. A popular existing approach, Reweighted Wake-Sleep (RWS), suffers from heavily biased gradients and a circular pathology that results in highly concentrated variational distributions. As an alternative, we propose SMC-Wake, a procedure for fitting an amortized variational approximation that uses likelihood-tempered sequential Monte Carlo samplers to estimate the gradient of the inclusive KL divergence. We propose three gradient estimators, all of which are asymptotically unbiased in the number of iterations and two of which are strongly consistent. Our method interleaves stochastic gradient updates, SMC samplers, and iterative improvement to an estimate of the normalizing constant to reduce bias from self-normalization. In experiments with both simulated and real datasets, SMC-Wake fits variational distributions that approximate the posterior more accurately than existing methods. △ Less

Submitted 15 March, 2024; originally announced March 2024.

Comments: Accepted to the International Conference on Artificial Intelligence and Statistics (AISTATS 2024)

Choose Your Own Adventure: Interactive E-Books to Improve Word Knowledge and Comprehension Skills

Authors: Stephanie Day, Jin K. Hwang, Tracy Arner, Danielle McNamara, Carol Connor

Abstract: The purpose of this feasibility study was to examine the potential impact of reading digital interactive e-books on essential skills that support reading comprehension with third-fifth grade students. Students read two e-Books that taught word learning and comprehension monitoring strategies in the service of learning difficult vocabulary and targeted science concepts about hurricanes. We investig… ▽ More The purpose of this feasibility study was to examine the potential impact of reading digital interactive e-books on essential skills that support reading comprehension with third-fifth grade students. Students read two e-Books that taught word learning and comprehension monitoring strategies in the service of learning difficult vocabulary and targeted science concepts about hurricanes. We investigated whether specific comprehension strategies including word learning and strategies that supported general reading comprehension, summarization, and question generation, show promise of effectiveness in building vocabulary knowledge and comprehension skills in the e-Books. Students were assigned to read one of three versions of each of the e-Books, each version implemented one strategy. The books employed a choose-your-adventure format with embedded comprehension questions that provided students with immediate feedback on their responses. Paired samples t-tests were run to examine pre-to-post differences in learning the targeted vocabulary and science concepts taught in both e-Books. For both e-Books, students demonstrated significant gains in word learning and on the targeted hurricane concepts. Additionally, Hierarchical Linear Modeling (HLM) revealed that no one strategy was more associated with larger gains than the other. Performance on the embedded questions in the books was also associated with greater posttest outcomes for both e-Books. This work discusses important considerations for implementation and future development of e-books that can enhance student engagement and improve reading comprehension. △ Less

Submitted 4 March, 2024; originally announced March 2024.

Variational Inference with Coverage Guarantees in Simulation-Based Inference

Authors: Yash Patel, Declan McNamara, Jackson Loper, Jeffrey Regier, Ambuj Tewari

Abstract: Amortized variational inference is an often employed framework in simulation-based inference that produces a posterior approximation that can be rapidly computed given any new observation. Unfortunately, there are few guarantees about the quality of these approximate posteriors. We propose Conformalized Amortized Neural Variational Inference (CANVI), a procedure that is scalable, easily implemente… ▽ More Amortized variational inference is an often employed framework in simulation-based inference that produces a posterior approximation that can be rapidly computed given any new observation. Unfortunately, there are few guarantees about the quality of these approximate posteriors. We propose Conformalized Amortized Neural Variational Inference (CANVI), a procedure that is scalable, easily implemented, and provides guaranteed marginal coverage. Given a collection of candidate amortized posterior approximators, CANVI constructs conformalized predictors based on each candidate, compares the predictors using a metric known as predictive efficiency, and returns the most efficient predictor. CANVI ensures that the resulting predictor constructs regions that contain the truth with a user-specified level of probability. CANVI is agnostic to design decisions in formulating the candidate approximators and only requires access to samples from the forward model, permitting its use in likelihood-free settings. We prove lower bounds on the predictive efficiency of the regions produced by CANVI and explore how the quality of a posterior approximation relates to the predictive efficiency of prediction regions based on that approximation. Finally, we demonstrate the accurate calibration and high predictive efficiency of CANVI on a suite of simulation-based inference benchmark tasks and an important scientific task: analyzing galaxy emission spectra. △ Less

Submitted 25 July, 2024; v1 submitted 23 May, 2023; originally announced May 2023.

Provably Fair Representations

Authors: Daniel McNamara, Cheng Soon Ong, Robert C. Williamson

Abstract: Machine learning systems are increasingly used to make decisions about people's lives, such as whether to give someone a loan or whether to interview someone for a job. This has led to considerable interest in making such machine learning systems fair. One approach is to transform the input data used by the algorithm. This can be achieved by passing each input data point through a representation f… ▽ More Machine learning systems are increasingly used to make decisions about people's lives, such as whether to give someone a loan or whether to interview someone for a job. This has led to considerable interest in making such machine learning systems fair. One approach is to transform the input data used by the algorithm. This can be achieved by passing each input data point through a representation function prior to its use in training or testing. Techniques for learning such representation functions from data have been successful empirically, but typically lack theoretical fairness guarantees. We show that it is possible to prove that a representation function is fair according to common measures of both group and individual fairness, as well as useful with respect to a target task. These provable properties can be used in a governance model involving a data producer, a data user and a data regulator, where there is a separation of concerns between fairness and target task utility to ensure transparency and prevent perverse incentives. We formally define the 'cost of mistrust' of using this model compared to the setting where there is a single trusted party, and provide bounds on this cost in particular cases. We present a practical approach to learning fair representation functions and apply it to financial and criminal justice datasets. We evaluate the fairness and utility of these representation functions using measures motivated by our theoretical results. △ Less

Submitted 12 October, 2017; originally announced October 2017.

A Modular Theory of Feature Learning

Authors: Daniel McNamara, Cheng Soon Ong, Robert C. Williamson

Abstract: Learning representations of data, and in particular learning features for a subsequent prediction task, has been a fruitful area of research delivering impressive empirical results in recent years. However, relatively little is understood about what makes a representation `good'. We propose the idea of a risk gap induced by representation learning for a given prediction context, which measures the… ▽ More Learning representations of data, and in particular learning features for a subsequent prediction task, has been a fruitful area of research delivering impressive empirical results in recent years. However, relatively little is understood about what makes a representation `good'. We propose the idea of a risk gap induced by representation learning for a given prediction context, which measures the difference in the risk of some learner using the learned features as compared to the original inputs. We describe a set of sufficient conditions for unsupervised representation learning to provide a benefit, as measured by this risk gap. These conditions decompose the problem of when representation learning works into its constituent parts, which can be separately evaluated using an unlabeled sample, suitable domain-specific assumptions about the joint distribution, and analysis of the feature learner and subsequent supervised learner. We provide two examples of such conditions in the context of specific properties of the unlabeled distribution, namely when the data lies close to a low-dimensional manifold and when it forms clusters. We compare our approach to a recently proposed analysis of semi-supervised learning. △ Less

Submitted 9 November, 2016; originally announced November 2016.

Gamed-based iSTART Practice: From MiBoard to Self-Explanation Showdown

Authors: Justin F. Brunelle, G. Tanner Jackson, Kyle Dempsey, Chutima Boonthum, Irwin B. Levinstein, Danielle S. McNamara

Abstract: MiBoard (Multiplayer Interactive Board Game) is an online, turnbased board game that was developed to assess the integration of game characteristics (point rewards, game-like interaction, and peer feedback) and how that might affect student engagement and learning efficacy. This online board game was designed to fit within the Extended Practice module of iSTART (Interactive Strategy Training for A… ▽ More MiBoard (Multiplayer Interactive Board Game) is an online, turnbased board game that was developed to assess the integration of game characteristics (point rewards, game-like interaction, and peer feedback) and how that might affect student engagement and learning efficacy. This online board game was designed to fit within the Extended Practice module of iSTART (Interactive Strategy Training for Active Reading and Thinking). Unfortunately, preliminary research shows that MiBoard actually reduces engagement and does not benefit the quality of student self-explanations when compared to the original Extended Practice module. Consequently the MiBoard framework has been revamped to create Self-Explanation Showdown, a faster-paced, less analytically oriented game that adds competition to the creation of self-explanations. Students are evaluated on the quality of their self-explanations using the same assessment algorithms from iSTART Extended Practice module (this includes both word-based and LSA-based assessments). The technical issues involved in development of MiBoard and Self- Explanation Showdown are described. The lessons learned from the MiBoard experience are also discussed in this paper. △ Less

Submitted 11 September, 2010; originally announced September 2010.

Journal ref: FLAIRS-23, May 22-23 2010

MiBoard: A Digital Game from a Physical World

Authors: Kyle B Dempsey, G. Tanner Jackson, Justin F. Brunelle, Michael Rowe, Danielle S. McNamara

Abstract: Increasing user engagement is constant challenge for Intelligent Tutoring Systems researchers. A current trend in the ITS field is to increase engagement of proven learning systems by integrating them within games, or adding in game like components. Incorporating proven learning methods within a game based environment is expected to add to the overall experience without detracting from the origina… ▽ More Increasing user engagement is constant challenge for Intelligent Tutoring Systems researchers. A current trend in the ITS field is to increase engagement of proven learning systems by integrating them within games, or adding in game like components. Incorporating proven learning methods within a game based environment is expected to add to the overall experience without detracting from the original goals, however, the current study demonstrates two important issues with regard to ITS design. First, effective designs from the physical world do not always translate into the digital world. Second, games do not necessarily improve engagement, and in some cases, they may have the opposite effect. The current study discusses the development and a brief assessment of MiBoard a multiplayer collaborative online board game designed to closely emulate a previously developed physical board game, iSTART: The Board Game. △ Less

Submitted 11 September, 2010; originally announced September 2010.

Journal ref: FLAIRS-23, May 22-23 2010

MiBoard: Multiplayer Interactive Board Game

Authors: Kyle B. Dempsey, Justin F. Brunelle, G. Tanner Jackson, Chutima Boonthum, Irwin B. Levinstein, Danielle S. McNamara

Abstract: Serious games have recently emerged as an avenue for curriculum delivery. Serious games incorporate motivation and entertainment while providing pointed curriculum for the user. This paper presents a serious game, called MiBoard, currently being developed from the iSTART Intelligent Tutoring System. MiBoard incorporates a multiplayer interaction that iSTART was previously unable to provide. This m… ▽ More Serious games have recently emerged as an avenue for curriculum delivery. Serious games incorporate motivation and entertainment while providing pointed curriculum for the user. This paper presents a serious game, called MiBoard, currently being developed from the iSTART Intelligent Tutoring System. MiBoard incorporates a multiplayer interaction that iSTART was previously unable to provide. This multiplayer interaction produces a wide variation across game trials, while also increasing the repeat playability for users. This paper presents a demonstration of the MiBoard system and the expectations for its application. △ Less

Submitted 11 September, 2010; originally announced September 2010.

Comments: 14th International Conference on Artificial Intelligence in Education (AIED), 2009

MiBoard: iSTART Metacognitive Training through Gaming

Authors: Justin F. Brunelle, Kyle B. Dempsey, G. Tanner Jackson, Chutima Boonthum, Irwin B. Levinstein, Danielle S. McNamara

Abstract: MiBoard (Multiplayer Interactive Board Game) is an online, turn-based board game, which is a supplement of the iSTART (Interactive Strategy Training for Active Reading and Thinking) application. MiBoard is developed to test the hypothesis that integrating game characteristics (point rewards, game-like interaction, and peer feedback) into the iSTART trainer will significantly improve its effectiven… ▽ More MiBoard (Multiplayer Interactive Board Game) is an online, turn-based board game, which is a supplement of the iSTART (Interactive Strategy Training for Active Reading and Thinking) application. MiBoard is developed to test the hypothesis that integrating game characteristics (point rewards, game-like interaction, and peer feedback) into the iSTART trainer will significantly improve its effectiveness on students' learning. It was shown by M. Rowe that a physical board game did in fact enhance students' performance. MiBoard is a computer-based version of Rowe's board game that eliminates constraints on locality while retaining the crucial practice components that were the game's objective. MiBoard gives incentives for participation and provides a more enjoyable and social practice environment compared to the online individual practice component of the original trainer △ Less

Submitted 11 September, 2010; originally announced September 2010.

Comments: SCiP Conference, November 19th 2009