-
Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models
Authors:
Sima Noorani,
Shayan Kiyani,
George Pappas,
Hamed Hassani
Abstract:
Uncertainty quantification (UQ) is essential for safe deployment of generative AI models such as large language models (LLMs), especially in high stakes applications. Conformal prediction (CP) offers a principled uncertainty quantification framework, but classical methods focus on regression and classification, relying on geometric distances or softmax scores: tools that presuppose structured outp…
▽ More
Uncertainty quantification (UQ) is essential for safe deployment of generative AI models such as large language models (LLMs), especially in high stakes applications. Conformal prediction (CP) offers a principled uncertainty quantification framework, but classical methods focus on regression and classification, relying on geometric distances or softmax scores: tools that presuppose structured outputs. We depart from this paradigm by studying CP in a query only setting, where prediction sets must be constructed solely from finite queries to a black box generative model, introducing a new trade off between coverage, test time query budget, and informativeness. We introduce Conformal Prediction with Query Oracle (CPQ), a framework characterizing the optimal interplay between these objectives. Our finite sample algorithm is built on two core principles: one governs the optimal query policy, and the other defines the optimal mapping from queried samples to prediction sets. Remarkably, both are rooted in the classical missing mass problem in statistics. Specifically, the optimal query policy depends on the rate of decay, or the derivative, of the missing mass, for which we develop a novel estimator. Meanwhile, the optimal mapping hinges on the missing mass itself, which we estimate using Good Turing estimators. We then turn our focus to implementing our method for language models, where outputs are vast, variable, and often under specified. Fine grained experiments on three real world open ended tasks and two LLMs, show CPQ applicability to any black box LLM and highlight: (1) individual contribution of each principle to CPQ performance, and (2) CPQ ability to yield significantly more informative prediction sets than existing conformal methods for language uncertainty quantification.
△ Less
Submitted 5 June, 2025;
originally announced June 2025.
-
On the cosine similarity and orthogonality between persistence diagrams
Authors:
Azmeer Nordin,
Mohd Salmi Md Noorani,
Nurulkamal Masseran,
Mohd Sabri Ismail,
Nur Firyal Roslan
Abstract:
Topological data analysis is an approach to study shape of a data set by means of topology. Its main object of study is the persistence diagram, which represents the topological features of the data set at different spatial resolutions. Multiple data sets can be compared by the similarity of their diagrams to understand their behaviors in relative to each other. The bottleneck and Wasserstein dist…
▽ More
Topological data analysis is an approach to study shape of a data set by means of topology. Its main object of study is the persistence diagram, which represents the topological features of the data set at different spatial resolutions. Multiple data sets can be compared by the similarity of their diagrams to understand their behaviors in relative to each other. The bottleneck and Wasserstein distances are often used as a tool to indicate the similarity. In this paper, we introduce cosine similarity as a new indicator for the similarity between persistence diagrams and investigate its properties. Furthermore, it leads to the new notion of orthogonality between persistence diagrams. It turns out that the orthogonality refers to perfect dissimilarity between persistence diagrams under the cosine similarity. Through data demonstration, the cosine similarity is shown to be more accurate than the standard distances to measure the similarity between persistence diagrams.
△ Less
Submitted 6 April, 2025;
originally announced April 2025.
-
Conformal Risk Minimization with Variance Reduction
Authors:
Sima Noorani,
Orlando Romero,
Nicolo Dal Fabbro,
Hamed Hassani,
George J. Pappas
Abstract:
Conformal prediction (CP) is a distribution-free framework for achieving probabilistic guarantees on black-box models. CP is generally applied to a model post-training. Recent research efforts, on the other hand, have focused on optimizing CP efficiency during training. We formalize this concept as the problem of conformal risk minimization (CRM). In this direction, conformal training (ConfTr) by…
▽ More
Conformal prediction (CP) is a distribution-free framework for achieving probabilistic guarantees on black-box models. CP is generally applied to a model post-training. Recent research efforts, on the other hand, have focused on optimizing CP efficiency during training. We formalize this concept as the problem of conformal risk minimization (CRM). In this direction, conformal training (ConfTr) by Stutz et al.(2022) is a technique that seeks to minimize the expected prediction set size of a model by simulating CP in-between training updates. Despite its potential, we identify a strong source of sample inefficiency in ConfTr that leads to overly noisy estimated gradients, introducing training instability and limiting practical use. To address this challenge, we propose variance-reduced conformal training (VR-ConfTr), a CRM method that incorporates a variance reduction technique in the gradient estimation of the ConfTr objective function. Through extensive experiments on various benchmark datasets, we demonstrate that VR-ConfTr consistently achieves faster convergence and smaller prediction sets compared to baselines.
△ Less
Submitted 8 February, 2025; v1 submitted 3 November, 2024;
originally announced November 2024.
-
Fostering Peer Learning through a New Game-Theoretical Approach in a Blended Learning Environment
Authors:
Seyede Fatemeh Noorani,
Mohammad Hossein Manshaei,
Mohammad Ali Montazeri,
Behnaz Omoomi
Abstract:
Obtaining knowledge and skill achievement through peer learning can lead to higher academic achievement. However, peer learning implementation is not just about putting students together and hoping for the best. At its worst-designed, peer learning may result in one person doing all the effort for instance, or may fail to encourage the students to interact enough with the task and so enhance the t…
▽ More
Obtaining knowledge and skill achievement through peer learning can lead to higher academic achievement. However, peer learning implementation is not just about putting students together and hoping for the best. At its worst-designed, peer learning may result in one person doing all the effort for instance, or may fail to encourage the students to interact enough with the task and so enhance the task in hand. This study proposes a mechanism as well as an instructional design to foster well-organized peer learning based on game theory $(PD\_PL)$. The proposed mechanism uses prisoner's dilemma and maps the strategy and payoff concepts found in prisoner's dilemma onto a peer learning atmosphere. PD\_PL was implemented during several sessions of four university courses and with 142 computer engineering students. %The results of the pre-test and post-test exams of all the sessions were compared with R software through Paired Hotelling's T-Square analysis in order to investigate the impacts of $PD\_PL$ and the proposed instructional design on students' personal learning. The study results indicated that PD\_PL was beneficial and favourable to the students.
Further analysis showed that the $PD\_PL$ had sometimes even enhanced learning by up to $47.2\%$.
%The results of a subjective evaluation showed that the majority of respondents found $PD\_PL$ to be an attractive and efficient tool for learning enhancement. %Everybody who is interested in designing peer learning programs and tools will find this study interesting.
△ Less
Submitted 27 October, 2019;
originally announced October 2019.
