-
Adaptive $k$-nearest neighbor classifier based on the local estimation of the shape operator
Authors:
Alexandre Luís Magalhães Levada,
Frank Nielsen,
Michel Ferreira Cardia Haddad
Abstract:
The $k$-nearest neighbor ($k$-NN) algorithm is one of the most popular methods for nonparametric classification. However, a relevant limitation concerns the definition of the number of neighbors $k$. This parameter exerts a direct impact on several properties of the classifier, such as the bias-variance tradeoff, smoothness of decision boundaries, robustness to noise, and class imbalance handling.…
▽ More
The $k$-nearest neighbor ($k$-NN) algorithm is one of the most popular methods for nonparametric classification. However, a relevant limitation concerns the definition of the number of neighbors $k$. This parameter exerts a direct impact on several properties of the classifier, such as the bias-variance tradeoff, smoothness of decision boundaries, robustness to noise, and class imbalance handling. In the present paper, we introduce a new adaptive $k$-nearest neighbours ($kK$-NN) algorithm that explores the local curvature at a sample to adaptively defining the neighborhood size. The rationale is that points with low curvature could have larger neighborhoods (locally, the tangent space approximates well the underlying data shape), whereas points with high curvature could have smaller neighborhoods (locally, the tangent space is a loose approximation). We estimate the local Gaussian curvature by computing an approximation to the local shape operator in terms of the local covariance matrix as well as the local Hessian matrix. Results on many real-world datasets indicate that the new $kK$-NN algorithm yields superior balanced accuracy compared to the established $k$-NN method and also another adaptive $k$-NN algorithm. This is particularly evident when the number of samples in the training data is limited, suggesting that the $kK$-NN is capable of learning more discriminant functions with less data considering many relevant cases.
△ Less
Submitted 8 September, 2024;
originally announced September 2024.
-
An information-geometric approach for network decomposition using the q-state Potts model
Authors:
Alexandre L. M. Levada
Abstract:
Complex networks are critical in many scientific, technological, and societal contexts due to their ability to represent and analyze intricate systems with interdependent components. Often, after labeling the nodes of a network with a community detection algorithm, its modular organization emerges, allowing a better understanding of the underlying structure by uncovering hidden relationships. In t…
▽ More
Complex networks are critical in many scientific, technological, and societal contexts due to their ability to represent and analyze intricate systems with interdependent components. Often, after labeling the nodes of a network with a community detection algorithm, its modular organization emerges, allowing a better understanding of the underlying structure by uncovering hidden relationships. In this paper, we introduce a novel information-geometric framework for the filtering and decomposition of networks whose nodes have been labeled. Our approach considers the labeled network as the outcome of a Markov random field modeled by a q-state Potts model. According to information geometry, the first and second order Fisher information matrices are related to the metric and curvature tensor of the parametric space of a statistical model. By computing an approximation to the local shape operator, the proposed methodology is able to identify low and high information nodes, allowing the decomposition of the labeled network in two complementary subgraphs. Hence, we call this method as the LO-HI decomposition. Experimental results with several kinds of networks show that the high information subgraph is often related to edges and boundaries, while the low information subgraph is a smoother version of the network, in the sense that the modular structure is improved.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
The geodesic dispersion phenomenon in random fields dynamics
Authors:
Alexandre L. M. Levada
Abstract:
Random fields are ubiquitous mathematical structures in physics, with applications ranging from thermodynamics and statistical physics to quantum field theory and cosmology. Recent works on information geometry of Gaussian random fields proposed mathematical expressions for the components of the metric tensor of the underlying parametric space, allowing the computation of the curvature in each poi…
▽ More
Random fields are ubiquitous mathematical structures in physics, with applications ranging from thermodynamics and statistical physics to quantum field theory and cosmology. Recent works on information geometry of Gaussian random fields proposed mathematical expressions for the components of the metric tensor of the underlying parametric space, allowing the computation of the curvature in each point of the manifold. In this study, our hypothesis is that time irreversibility in Gaussian random fields dynamics is a direct consequence of intrinsic geometric properties (curvature) of their parametric space. In order to validate this hypothesis, we compute the components of the metric tensor and derive the twenty seven Christoffel symbols of the metric to define the Euler-Lagrange equations, a system of partial differential equations that are used to build geodesic curves in Riemannian manifolds. After that, by the application of the fourth-order Runge-Kutta method and Markov Chain Monte Carlo simulation, we numerically build geodesic curves starting from an arbitrary initial point in the manifold. The obtained results show that, when the system undergoes phase transitions, the geodesic curve obtained by time reversing the computational simulation diverges from the original curve, showing a strange effect that we called the geodesic dispersion phenomenon, which suggests that time irreversibility in random fields is related to the intrinsic geometry of their parametric space.
△ Less
Submitted 2 May, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
On the Kullback-Leibler divergence between pairwise isotropic Gaussian-Markov random fields
Authors:
Alexandre L. M. Levada
Abstract:
The Kullback-Leibler divergence or relative entropy is an information-theoretic measure between statistical models that play an important role in measuring a distance between random variables. In the study of complex systems, random fields are mathematical structures that models the interaction between these variables by means of an inverse temperature parameter, responsible for controlling the sp…
▽ More
The Kullback-Leibler divergence or relative entropy is an information-theoretic measure between statistical models that play an important role in measuring a distance between random variables. In the study of complex systems, random fields are mathematical structures that models the interaction between these variables by means of an inverse temperature parameter, responsible for controlling the spatial dependence structure along the field. In this paper, we derive closed-form expressions for the Kullback-Leibler divergence between two pairwise isotropic Gaussian-Markov random fields in both univariate and multivariate cases. The proposed equation allows the development of novel similarity measures in image processing and machine learning applications, such as image denoising and unsupervised metric learning.
△ Less
Submitted 24 March, 2022;
originally announced March 2022.
-
The Curvature Effect in Gaussian Random Fields
Authors:
Alexandre L. M. Levada
Abstract:
Random field models are mathematical structures used in the study of stochastic complex systems. In this paper, we compute the shape operator of Gaussian random field manifolds using the first and second fundamental forms (Fisher information matrices). Using Markov Chain Monte Carlo techniques, we simulate the dynamics of these random fields and compute the Gaussian curvature of the parametric spa…
▽ More
Random field models are mathematical structures used in the study of stochastic complex systems. In this paper, we compute the shape operator of Gaussian random field manifolds using the first and second fundamental forms (Fisher information matrices). Using Markov Chain Monte Carlo techniques, we simulate the dynamics of these random fields and compute the Gaussian curvature of the parametric space, analyzing how this quantity changes along phase transitions. During the simulation, we have observed an unexpected phenomenon that we called the \emph{curvature effect}, which indicates that a highly asymmetric geometric deformation happens in the underlying parametric space when there are significant increase/decrease in the system's entropy. This asymmetric pattern relates to the emergence of hysteresis, leading to an intrinsic arrow of time along the dynamics.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
Geodesic curves in Gaussian random field manifolds
Authors:
Alexandre L. M. Levada
Abstract:
Random fields are mathematical structures used to model the spatial interaction of random variables along time, with applications ranging from statistical physics and thermodynamics to system's biology and the simulation of complex systems. Despite being studied since the 19th century, little is known about how the dynamics of random fields are related to the geometric properties of their parametr…
▽ More
Random fields are mathematical structures used to model the spatial interaction of random variables along time, with applications ranging from statistical physics and thermodynamics to system's biology and the simulation of complex systems. Despite being studied since the 19th century, little is known about how the dynamics of random fields are related to the geometric properties of their parametric spaces. For example, how can we quantify the similarity between two random fields operating in different regimes using an intrinsic measure? In this paper, we propose a numerical method for the computation of geodesic distances in Gaussian random field manifolds. First, we derive the metric tensor of the underlying parametric space (the 3 x 3 first-order Fisher information matrix), then we derive the 27 Christoffel symbols required in the definition of the system of non-linear differential equations whose solution is a geodesic curve starting at the initial conditions. The fourth-order Runge-Kutta method is applied to numerically solve the non-linear system through an iterative approach. The obtained results show that the proposed method can estimate the geodesic distances for several different initial conditions. Besides, the results reveal an interesting pattern: in several cases, the geodesic curve obtained by reversing the system of differential equations in time does not match the original curve, suggesting the existence of irreversible geometric deformations in the trajectory of a moving reference traveling along a geodesic curve.
△ Less
Submitted 6 November, 2021;
originally announced November 2021.
-
On the curvatures of Gaussian random field manifolds
Authors:
Alexandre L. M. Levada
Abstract:
Information geometry is concerned with the application of differential geometry concepts in the study of the parametric spaces of statistical models. When the random variables are independent and identically distributed, the underlying parametric space exhibit constant curvature, which makes the geometry hyperbolic (negative) or spherical (positive). In this paper, we derive closed-form expression…
▽ More
Information geometry is concerned with the application of differential geometry concepts in the study of the parametric spaces of statistical models. When the random variables are independent and identically distributed, the underlying parametric space exhibit constant curvature, which makes the geometry hyperbolic (negative) or spherical (positive). In this paper, we derive closed-form expressions for the components of the first and second fundamental forms regarding pairwise isotropic Gaussian-Markov random field manifolds, allowing the computation of the Gaussian, mean and principal curvatures. Computational simulations using Markov Chain Monte Carlo dynamics indicate that a change in the sign of the Gaussian curvature is related to the emergence of phase transitions in the field. Moreover, the curvatures are highly asymmetrical for positive and negative displacements in the inverse temperature parameter, suggesting the existence of irreversible geometric properties in the parametric space along the dynamics. Furthermore, these asymmetric changes in the curvature of the space induces an intrinsic notion of time in the evolution of the random field.
△ Less
Submitted 2 October, 2021; v1 submitted 19 September, 2021;
originally announced September 2021.
-
Information geometry, simulation and complexity in Gaussian random fields
Authors:
Alexandre L. M. Levada
Abstract:
Random fields are useful mathematical objects in the characterization of non-deterministic complex systems. A fundamental issue in the evolution of dynamical systems is how intrinsic properties of such structures change in time. In this paper, we propose to quantify how changes in the spatial dependence structure affect the Riemannian metric tensor that equips the model's parametric space. Definin…
▽ More
Random fields are useful mathematical objects in the characterization of non-deterministic complex systems. A fundamental issue in the evolution of dynamical systems is how intrinsic properties of such structures change in time. In this paper, we propose to quantify how changes in the spatial dependence structure affect the Riemannian metric tensor that equips the model's parametric space. Defining Fisher curves, we measure the variations in each component of the metric tensor when visiting different entropic states of the system. Simulations show that the geometric deformations induced by the metric tensor in case of a decrease in the inverse temperature are not reversible for an increase of the same amount, provided there is significant variation in the system entropy: the process of taking a system from a lower entropy state A to a higher entropy state B and then bringing it back to A, induces a natural intrinsic one-way direction of evolution. In this context, Fisher curves resemble mathematical models of hysteresis in which the natural orientation is pointed by an arrow of time.
△ Less
Submitted 13 March, 2017;
originally announced March 2017.
-
Learning from Complex Systems: On the Roles of Entropy and Fisher Information in Pairwise Isotropic Gaussian Markov Random Fields
Authors:
Alexandre L. M. Levada
Abstract:
Markov Random Field models are powerful tools for the study of complex systems. However, little is known about how the interactions between the elements of such systems are encoded, especially from an information-theoretic perspective. In this paper, our goal is to enlight the connection between Fisher information, Shannon entropy, information geometry and the behavior of complex systems modeled b…
▽ More
Markov Random Field models are powerful tools for the study of complex systems. However, little is known about how the interactions between the elements of such systems are encoded, especially from an information-theoretic perspective. In this paper, our goal is to enlight the connection between Fisher information, Shannon entropy, information geometry and the behavior of complex systems modeled by isotropic pairwise Gaussian Markov random fields. We propose analytical expressions to compute local and global versions of these measures using Besag's pseudo-likelihood function, characterizing the system's behavior through its \emph{Fisher curve}, a parametric trajectory accross the information space that provides a geometric representation for the study of complex systems. Computational experiments show how the proposed tools can be useful in extrating relevant information from complex patterns. The obtained results quantify and support our main conclusion, which is: in terms of information, moving towards higher entropy states (A --> B) is different from moving towards lower entropy states (B --> A), since the \emph{Fisher curves} are not the same given a natural orientation (the direction of time).
△ Less
Submitted 16 October, 2013; v1 submitted 24 August, 2011;
originally announced August 2011.
