-
An approach to identify the most semantically informative deep representations of text and images
Authors:
Santiago Acevedo,
Andrea Mascaretti,
Riccardo Rende,
Matéo Mahaut,
Marco Baroni,
Alessandro Laio
Abstract:
Deep neural networks are known to develop similar representations for semantically related data, even when they belong to different domains, such as an image and its description, or the same text in different languages. We present a method for quantitatively investigating this phenomenon by measuring the relative information content of the representations of semantically related data and probing h…
▽ More
Deep neural networks are known to develop similar representations for semantically related data, even when they belong to different domains, such as an image and its description, or the same text in different languages. We present a method for quantitatively investigating this phenomenon by measuring the relative information content of the representations of semantically related data and probing how it is encoded into multiple tokens of large language models (LLMs) and vision transformers. Looking first at how LLMs process pairs of translated sentences, we identify inner ``semantic'' layers containing the most language-transferable information. We find moreover that, on these layers, a larger LLM (DeepSeek-V3) extracts significantly more general information than a smaller one (Llama3.1-8B). Semantic information is spread across many tokens and it is characterized by long-distance correlations between tokens and by a causal left-to-right (i.e., past-future) asymmetry. We also identify layers encoding semantic information within visual transformers. We show that caption representations in the semantic layers of LLMs predict visual representations of the corresponding images. We observe significant and model-dependent information asymmetries between image and text representations.
△ Less
Submitted 21 May, 2025;
originally announced May 2025.
-
Unsupervised detection of semantic correlations in big data
Authors:
Santiago Acevedo,
Alex Rodriguez,
Alessandro Laio
Abstract:
In real-world data, information is stored in extremely large feature vectors. These variables are typically correlated due to complex interactions involving many features simultaneously. Such correlations qualitatively correspond to semantic roles and are naturally recognized by both the human brain and artificial neural networks. This recognition enables, for instance, the prediction of missing p…
▽ More
In real-world data, information is stored in extremely large feature vectors. These variables are typically correlated due to complex interactions involving many features simultaneously. Such correlations qualitatively correspond to semantic roles and are naturally recognized by both the human brain and artificial neural networks. This recognition enables, for instance, the prediction of missing parts of an image or text based on their context. We present a method to detect these correlations in high-dimensional data represented as binary numbers. We estimate the binary intrinsic dimension of a dataset, which quantifies the minimum number of independent coordinates needed to describe the data, and is therefore a proxy of semantic complexity. The proposed algorithm is largely insensitive to the so-called curse of dimensionality, and can therefore be used in big data analysis. We test this approach identifying phase transitions in model magnetic systems and we then apply it to the detection of semantic correlations of images and text inside deep neural networks.
△ Less
Submitted 21 May, 2025; v1 submitted 4 November, 2024;
originally announced November 2024.
-
Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations
Authors:
Lorenzo Basile,
Santiago Acevedo,
Luca Bortolussi,
Fabio Anselmi,
Alex Rodriguez
Abstract:
To gain insight into the mechanisms behind machine learning methods, it is crucial to establish connections among the features describing data points. However, these correlations often exhibit a high-dimensional and strongly nonlinear nature, which makes them challenging to detect using standard methods. This paper exploits the entanglement between intrinsic dimensionality and correlation to propo…
▽ More
To gain insight into the mechanisms behind machine learning methods, it is crucial to establish connections among the features describing data points. However, these correlations often exhibit a high-dimensional and strongly nonlinear nature, which makes them challenging to detect using standard methods. This paper exploits the entanglement between intrinsic dimensionality and correlation to propose a metric that quantifies the (potentially nonlinear) correlation between high-dimensional manifolds. We first validate our method on synthetic data in controlled environments, showcasing its advantages and drawbacks compared to existing techniques. Subsequently, we extend our analysis to large-scale applications in neural network representations. Specifically, we focus on latent representations of multimodal data, uncovering clear correlations between paired visual and textual embeddings, whereas existing methods struggle significantly in detecting similarity. Our results indicate the presence of highly nonlinear correlation patterns between latent manifolds.
△ Less
Submitted 2 March, 2025; v1 submitted 22 June, 2024;
originally announced June 2024.
-
Effects and Coeffects in Call-By-Push-Value (Extended Version)
Authors:
Cassia Torczon,
Emmanuel Suárez Acevedo,
Shubh Agrawal,
Joey Velez-Ginorio,
Stephanie Weirich
Abstract:
Effect and coeffect tracking integrate many types of compile-time analysis, such as cost, liveness, or dataflow, directly into a language's type system. In this paper, we investigate the addition of effect and coeffect tracking to the type system of call-by-push-value (CBPV), a computational model useful in compilation for its isolation of effects and for its ability to cleanly express both call-b…
▽ More
Effect and coeffect tracking integrate many types of compile-time analysis, such as cost, liveness, or dataflow, directly into a language's type system. In this paper, we investigate the addition of effect and coeffect tracking to the type system of call-by-push-value (CBPV), a computational model useful in compilation for its isolation of effects and for its ability to cleanly express both call-by-name and call-by-value computations. Our main result is effect-and-coeffect soundness, which asserts that the type system accurately bounds the effects that the program may trigger during execution and accurately tracks the demands that the program may make on its environment. This result holds for two different dynamic semantics: a generic one that can be adapted for different coeffects and one that is adapted for reasoning about resource usage. In particular, the second semantics discards the evaluation of unused values and pure computations while ensuring that effectful computations are always evaluated, even if their results are not required. Our results have been mechanized using the Coq proof assistant.
△ Less
Submitted 30 August, 2024; v1 submitted 20 November, 2023;
originally announced November 2023.
-
Making Logical Relations More Relatable (Proof Pearl)
Authors:
Emmanuel Suárez Acevedo,
Stephanie Weirich
Abstract:
Mechanical proofs by logical relations often involve tedious reasoning about substitution. In this paper, we show that this is not necessarily the case, by developing, in Agda, a proof that all simply typed lambda calculus expressions evaluate to values. A formalization of the proof is remarkably short (~40 lines of code), making for an excellent introduction to the technique of proofs by logical…
▽ More
Mechanical proofs by logical relations often involve tedious reasoning about substitution. In this paper, we show that this is not necessarily the case, by developing, in Agda, a proof that all simply typed lambda calculus expressions evaluate to values. A formalization of the proof is remarkably short (~40 lines of code), making for an excellent introduction to the technique of proofs by logical relations not only on paper but also in a mechanized setting. We then show that this process extends to more sophisticated reasoning by also proving the totality of normalization by evaluation. Although these proofs are not new, we believe presenting them will empower both new and experienced programming language theorists in their use of logical relations.
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
-
Damage detection in a unidimensional truss using the firefly optimization algorithm and finite elements
Authors:
Camilo Manrique Escobar,
Octavio Andrés González-Estrada,
Heller Guillermo Sánchez Acevedo
Abstract:
In this paper, we investigate the damage detection of structures seen as an optimization problem, using modal characterization to evaluate the dynamic response of the structure given a damage model. We implemented the firefly optimization algorithm with a simple numerical damage model to assess the performance of the method and its advantages for structural health monitoring (SHM). We show some im…
▽ More
In this paper, we investigate the damage detection of structures seen as an optimization problem, using modal characterization to evaluate the dynamic response of the structure given a damage model. We implemented the firefly optimization algorithm with a simple numerical damage model to assess the performance of the method and its advantages for structural health monitoring (SHM). We show some implementation details and discuss the obtained results.
△ Less
Submitted 24 April, 2018; v1 submitted 9 June, 2017;
originally announced June 2017.
