-
An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning
Authors:
Navdeep Kaur,
Lachlan McPheat,
Alessandra Russo,
Anthony G Cohn,
Pranava Madhyastha
Abstract:
In this paper, we examine the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP) to enhance the performance of standard open-weight LLMs on complex multi-step reasoning tasks. Using the StepGame dataset, which requires spatial reasoning, we apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs. Exper…
▽ More
In this paper, we examine the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP) to enhance the performance of standard open-weight LLMs on complex multi-step reasoning tasks. Using the StepGame dataset, which requires spatial reasoning, we apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs. Experimental results show that CLM significantly outperforms baseline models that use standard sampling methods, achieving substantial accuracy improvements across different levels of reasoning complexity. Additionally, the LLM-as-Judge metric enhances CLM's performance, especially in assessing structurally and logically correct ASP outputs. However, calibrating CLM with diverse calibration sets did not improve generalizability for tasks requiring much longer reasoning steps, indicating limitations in handling more complex tasks.
△ Less
Submitted 11 April, 2025; v1 submitted 7 March, 2025;
originally announced March 2025.
-
DisCoCat for Donkey Sentences
Authors:
Lachlan McPheat,
Daphne Wang
Abstract:
We demonstrate how to parse Geach's Donkey sentences in a compositional distributional model of meaning. We build on previous work on the DisCoCat (Distributional Compositional Categorical) framework, including extensions that model discourse, determiners, and relative pronouns. We present a type-logical syntax for parsing donkey sentences, for which we define both relational and vector space sem…
▽ More
We demonstrate how to parse Geach's Donkey sentences in a compositional distributional model of meaning. We build on previous work on the DisCoCat (Distributional Compositional Categorical) framework, including extensions that model discourse, determiners, and relative pronouns. We present a type-logical syntax for parsing donkey sentences, for which we define both relational and vector space semantics.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
A Quantum Natural Language Processing Approach to Pronoun Resolution
Authors:
Hadi Wazni,
Kin Ian Lo,
Lachlan McPheat,
Mehrnoosh Sadrzadeh
Abstract:
We use the Lambek Calculus with soft sub-exponential modalities to model and reason about discourse relations such as anaphora and ellipsis. A semantics for this logic is obtained by using truncated Fock spaces, developed in our previous work. We depict these semantic computations via a new string diagram. The Fock Space semantics has the advantage that its terms are learnable from large corpora o…
▽ More
We use the Lambek Calculus with soft sub-exponential modalities to model and reason about discourse relations such as anaphora and ellipsis. A semantics for this logic is obtained by using truncated Fock spaces, developed in our previous work. We depict these semantic computations via a new string diagram. The Fock Space semantics has the advantage that its terms are learnable from large corpora of data using machine learning and they can be experimented with on mainstream natural language tasks. Further, and thanks to an existing translation from vector spaces to quantum circuits, we can also learn these terms on quantum computers and their simulators, such as the IBMQ range. We extend the existing translation to Fock spaces and develop quantum circuit semantics for discourse relations. We then experiment with the IBMQ AerSimulations of these circuits in a definite pronoun resolution task, where the highest accuracies were recorded for models when the anaphora was resolved.
△ Less
Submitted 10 August, 2022;
originally announced August 2022.
-
Vector Space Semantics for Lambek Calculus with Soft Subexponentials
Authors:
Lachlan McPheat,
Hadi Wazni,
Mehrnoosh Sadrzadeh
Abstract:
We develop a vector space semantics for Lambek Calculus with Soft Subexponentials, apply the calculus to construct compositional vector interpretations for parasitic gap noun phrases and discourse units with anaphora and ellipsis, and experiment with the constructions in a distributional sentence similarity task. As opposed to previous work, which used Lambek Calculus with a Relevant Modality the…
▽ More
We develop a vector space semantics for Lambek Calculus with Soft Subexponentials, apply the calculus to construct compositional vector interpretations for parasitic gap noun phrases and discourse units with anaphora and ellipsis, and experiment with the constructions in a distributional sentence similarity task. As opposed to previous work, which used Lambek Calculus with a Relevant Modality the calculus used in this paper uses a bounded version of the modality and is decidable. The vector space semantics of this new modality allows us to meaningfully define contraction as projection and provide a linear theory behind what we could previously only achieve via nonlinear maps.
△ Less
Submitted 6 October, 2023; v1 submitted 22 November, 2021;
originally announced November 2021.
-
Anaphora and Ellipsis in Lambek Calculus with a Relevant Modality: Syntax and Semantics
Authors:
Lachlan McPheat,
Gijs Wijnholds,
Mehrnoosh Sadrzadeh,
Adriana Correia,
Alexis Toumi
Abstract:
Lambek calculus with a relevant modality $!\mathbf{L^*}$ of arXiv:1601.06303 syntactically resolves parasitic gaps in natural language. It resembles the Lambek calculus with anaphora $\mathbf{LA}$ of (Jäger, 1998) and the Lambek calculus with controlled contraction, $\mathbf{L}_{\Diamond}$, of arXiv:1905.01647v1 which deal with anaphora and ellipsis. What all these calculi add to Lambek calculus i…
▽ More
Lambek calculus with a relevant modality $!\mathbf{L^*}$ of arXiv:1601.06303 syntactically resolves parasitic gaps in natural language. It resembles the Lambek calculus with anaphora $\mathbf{LA}$ of (Jäger, 1998) and the Lambek calculus with controlled contraction, $\mathbf{L}_{\Diamond}$, of arXiv:1905.01647v1 which deal with anaphora and ellipsis. What all these calculi add to Lambek calculus is a copying and moving behaviour. Distributional semantics is a subfield of Natural Language Processing that uses vector space semantics for words via co-occurrence statistics in large corpora of data. Compositional vector space semantics for Lambek Calculi are obtained via the DisCoCat models arXiv:1003.4394v1. $\mathbf{LA}$ does not have a vector space semantics and the semantics of $\mathbf{L}_{\Diamond}$ is not compositional. Previously, we developed a DisCoCat semantics for $!\mathbf{L^*}$ and focused on the parasitic gap applications. In this paper, we use the vector space instance of that general semantics and show how one can also interpret anaphora, ellipsis, and for the first time derive the sloppy vs strict vector readings of ambiguous anaphora with ellipsis cases. The base of our semantics is tensor algebras and their finite dimensional variants: the Fermionic Fock spaces of Quantum Mechanics. We implement our model and experiment with the ellipsis disambiguation task of arXiv:1905.01647.
△ Less
Submitted 20 October, 2021;
originally announced October 2021.
-
Categorical Vector Space Semantics for Lambek Calculus with a Relevant Modality (Extended Abstract)
Authors:
Lachlan McPheat,
Mehrnoosh Sadrzadeh,
Hadi Wazni,
Gijs Wijnholds
Abstract:
We develop a categorical compositional distributional semantics for Lambek Calculus with a Relevant Modality, which has a limited version of the contraction and permutation rules. The categorical part of the semantics is a monoidal biclosed category with a coalgebra modality as defined on Differential Categories. We instantiate this category to finite dimensional vector spaces and linear maps via…
▽ More
We develop a categorical compositional distributional semantics for Lambek Calculus with a Relevant Modality, which has a limited version of the contraction and permutation rules. The categorical part of the semantics is a monoidal biclosed category with a coalgebra modality as defined on Differential Categories. We instantiate this category to finite dimensional vector spaces and linear maps via quantisation functors and work with three concrete interpretations of the coalgebra modality. We apply the model to construct categorical and concrete semantic interpretations for the motivating example of this extended calculus: the derivation of a phrase with a parasitic gap. The effectiveness of the concrete interpretations are evaluated via a disambiguation task, on an extension of a sentence disambiguation dataset to parasitic gap phrases, using BERT, Word2Vec, and FastText vectors and Relational tensors
△ Less
Submitted 25 January, 2021;
originally announced January 2021.
-
Categorical Vector Space Semantics for Lambek Calculus with a Relevant Modality
Authors:
Lachlan McPheat,
Mehrnoosh Sadrzadeh,
Hadi Wazni,
Gijs Wijnholds
Abstract:
We develop a categorical compositional distributional semantics for Lambek Calculus with a Relevant Modality !L*, which has a limited edition of the contraction and permutation rules. The categorical part of the semantics is a monoidal biclosed category with a coalgebra modality, very similar to the structure of a Differential Category. We instantiate this category to finite dimensional vector spa…
▽ More
We develop a categorical compositional distributional semantics for Lambek Calculus with a Relevant Modality !L*, which has a limited edition of the contraction and permutation rules. The categorical part of the semantics is a monoidal biclosed category with a coalgebra modality, very similar to the structure of a Differential Category. We instantiate this category to finite dimensional vector spaces and linear maps via "quantisation" functors and work with three concrete interpretations of the coalgebra modality. We apply the model to construct categorical and concrete semantic interpretations for the motivating example of !L*: the derivation of a phrase with a parasitic gap. The effectiveness of the concrete interpretations are evaluated via a disambiguation task, on an extension of a sentence disambiguation dataset to parasitic gap phrases, using BERT, Word2Vec, and FastText vectors and Relational tensors.
△ Less
Submitted 11 May, 2023; v1 submitted 6 May, 2020;
originally announced May 2020.
