-
C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation
Authors:
Yanyang Li,
Tin Long Wong,
Cheung To Hung,
Jianqiao Zhao,
Duo Zheng,
Ka Wai Liu,
Michael R. Lyu,
Liwei Wang
Abstract:
Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompass…
▽ More
Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C$^2$LEVA.
△ Less
Submitted 29 May, 2025; v1 submitted 6 December, 2024;
originally announced December 2024.
-
Information Geometry for the Working Information Theorist
Authors:
Kumar Vijay Mishra,
M. Ashok Kumar,
Ting-Kam Leonard Wong
Abstract:
Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas…
▽ More
Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.
△ Less
Submitted 5 October, 2023;
originally announced October 2023.
-
Tsallis and Rényi deformations linked via a new $λ$-duality
Authors:
Ting-Kam Leonard Wong,
Jun Zhang
Abstract:
Tsallis and Rényi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the $q$-exponential family which has applications in non-extensive statistical physics, information theory and statistics. In previous information-geometric studies, the $q$-exponential fami…
▽ More
Tsallis and Rényi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the $q$-exponential family which has applications in non-extensive statistical physics, information theory and statistics. In previous information-geometric studies, the $q$-exponential family was analyzed using classical convex duality and Bregman divergence. In this paper, we show that a generalized $λ$-duality, where $λ= 1 - q$ is the constant information-geometric curvature, leads to a generalized exponential family which is essentially equivalent to the $q$-exponential family and has deep connections with Rényi entropy and optimal transport. Using this generalized convex duality and its associated logarithmic divergence, we show that our $λ$-exponential family satisfies properties that parallel and generalize those of the exponential family. Under our framework, the Rényi entropy and divergence arise naturally, and we give a new proof of the Tsallis/Rényi entropy maximizing property of the $q$-exponential family. We also introduce a $λ$-mixture family which may be regarded as the dual of the $λ$-exponential family, and connect it with other mixture-type families. Finally, we discuss a duality between the $λ$-exponential family and the $λ$-logarithmic divergence, and study its statistical consequences.
△ Less
Submitted 12 January, 2022; v1 submitted 25 July, 2021;
originally announced July 2021.
-
Scalable Gradients for Stochastic Differential Equations
Authors:
Xuechen Li,
Ting-Kam Leonard Wong,
Ricky T. Q. Chen,
David Duvenaud
Abstract:
The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differential equation whose solution is the gradient, a memory-efficient algorithm for c…
▽ More
The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differential equation whose solution is the gradient, a memory-efficient algorithm for caching noise, and conditions under which numerical solutions converge. In addition, we combine our method with gradient-based stochastic variational inference for latent stochastic differential equations. We use our method to fit stochastic dynamics defined by neural networks, achieving competitive performance on a 50-dimensional motion capture dataset.
△ Less
Submitted 18 October, 2020; v1 submitted 5 January, 2020;
originally announced January 2020.
-
Logarithmic divergences from optimal transport and Rényi geometry
Authors:
Ting-Kam Leonard Wong
Abstract:
Divergences, also known as contrast functions, are distance-like quantities defined on manifolds of non-negative or probability measures. Using the duality in optimal transport, we introduce and study the one-parameter family of $L^{(\pm α)}$-divergences. It includes the Bregman divergence corresponding to the Euclidean quadratic cost, and the $L$-divergence introduced by Pal and the author in con…
▽ More
Divergences, also known as contrast functions, are distance-like quantities defined on manifolds of non-negative or probability measures. Using the duality in optimal transport, we introduce and study the one-parameter family of $L^{(\pm α)}$-divergences. It includes the Bregman divergence corresponding to the Euclidean quadratic cost, and the $L$-divergence introduced by Pal and the author in connection with portfolio theory and a logarithmic cost function. They admit natural generalizations of exponential family that are closely related to the $α$-family and $q$-exponential family. In particular, the $L^{(\pm α)}$-divergences of the corresponding potential functions are Rényi divergences. Using this unified framework we prove that the induced geometries are dually projectively flat with constant sectional curvatures, and a generalized Pythagorean theorem holds true. Conversely, we show that if a statistical manifold is dually projectively flat with constant curvature $\pm α$ with $α> 0$, then it is locally induced by an $L^{(\mp α)}$-divergence. We define in this context a canonical divergence which extends the one for dually flat manifolds.
△ Less
Submitted 3 September, 2018; v1 submitted 10 December, 2017;
originally announced December 2017.
-
Some observations on the logical foundations of inductive theorem proving
Authors:
Stefan Hetzl,
Tin Lok Wong
Abstract:
In this paper we study the logical foundations of automated inductive theorem proving. To that aim we first develop a theoretical model that is centered around the difficulty of finding induction axioms which are sufficient for proving a goal.
Based on this model, we then analyze the following aspects: the choice of a proof shape, the choice of an induction rule and the language of the induction…
▽ More
In this paper we study the logical foundations of automated inductive theorem proving. To that aim we first develop a theoretical model that is centered around the difficulty of finding induction axioms which are sufficient for proving a goal.
Based on this model, we then analyze the following aspects: the choice of a proof shape, the choice of an induction rule and the language of the induction formula. In particular, using model-theoretic techniques, we clarify the relationship between notions of inductiveness that have been considered in the literature on automated inductive theorem proving. This is a corrected version of the paper arXiv:1704.01930v5 published originally on Nov.~16, 2017.
△ Less
Submitted 12 April, 2018; v1 submitted 6 April, 2017;
originally announced April 2017.
-
Exponentially concave functions and a new information geometry
Authors:
Soumik Pal,
Ting-Kam Leonard Wong
Abstract:
A function is exponentially concave if its exponential is concave. We consider exponentially concave functions on the unit simplex. In a previous paper we showed that gradient maps of exponentially concave functions provide solutions to a Monge-Kantorovich optimal transport problem and give a better gradient approximation than those of ordinary concave functions. The approximation error, called L-…
▽ More
A function is exponentially concave if its exponential is concave. We consider exponentially concave functions on the unit simplex. In a previous paper we showed that gradient maps of exponentially concave functions provide solutions to a Monge-Kantorovich optimal transport problem and give a better gradient approximation than those of ordinary concave functions. The approximation error, called L-divergence, is different from the usual Bregman divergence. Using tools of information geometry and optimal transport, we show that L-divergence induces a new information geometry on the simplex consisting of a Riemannian metric and a pair of dually coupled affine connections which defines two kinds of geodesics. We show that the induced geometry is dually projectively flat but not flat. Nevertheless, we prove an analogue of the celebrated generalized Pythagorean theorem from classical information geometry. On the other hand, we consider displacement interpolation under a Lagrangian integral action that is consistent with the optimal transport problem and show that the action minimizing curves are dual geodesics. The Pythagorean theorem is also shown to have an interesting application of determining the optimal trading frequency in stochastic portfolio theory.
△ Less
Submitted 31 May, 2017; v1 submitted 19 May, 2016;
originally announced May 2016.
