Showing 1–13 of 13 results for author: Brunel, J

Towards a rigorous evaluation of RAG systems: the challenge of due diligence

Authors: Grégoire Martinon, Alexandra Lorenzo de Brionne, Jérôme Bohard, Antoine Lojou, Damien Hervault, Nicolas J-B. Brunel

Abstract: The rise of generative AI, has driven significant advancements in high-risk sectors like healthcare and finance. The Retrieval-Augmented Generation (RAG) architecture, combining language models (LLMs) with search engines, is particularly notable for its ability to generate responses from document corpora. Despite its potential, the reliability of RAG systems in critical contexts remains a concern,… ▽ More The rise of generative AI, has driven significant advancements in high-risk sectors like healthcare and finance. The Retrieval-Augmented Generation (RAG) architecture, combining language models (LLMs) with search engines, is particularly notable for its ability to generate responses from document corpora. Despite its potential, the reliability of RAG systems in critical contexts remains a concern, with issues such as hallucinations persisting. This study evaluates a RAG system used in due diligence for an investment fund. We propose a robust evaluation protocol combining human annotations and LLM-Judge annotations to identify system failures, like hallucinations, off-topic, failed citations, and abstentions. Inspired by the Prediction Powered Inference (PPI) method, we achieve precise performance measurements with statistical guarantees. We provide a comprehensive dataset for further analysis. Our contributions aim to enhance the reliability and scalability of RAG systems evaluation protocols in industrial applications. △ Less

Submitted 29 July, 2025; originally announced July 2025.

Comments: in French language. EvalLLM2025: Workshop on Evaluation Generative Models (LLM) and Challenges, AMIAD, 2025, Marseille, France

Efficient Precision Control in Object Detection Models for Enhanced and Reliable Ovarian Follicle Counting

Authors: Vincent Blot, Alexandra Lorenzo de Brionne, Ines Sellami, Olivier Trassard, Isabelle Beau, Charlotte Sonigo, Nicolas J-B. Brunel

Abstract: Image analysis is a key tool for describing the detailed mechanisms of folliculogenesis, such as evaluating the quantity of mouse Primordial ovarian Follicles (PMF) in the ovarian reserve. The development of high-resolution virtual slide scanners offers the possibility of quantifying, robustifying and accelerating the histopathological procedure. A major challenge for machine learning is to contro… ▽ More Image analysis is a key tool for describing the detailed mechanisms of folliculogenesis, such as evaluating the quantity of mouse Primordial ovarian Follicles (PMF) in the ovarian reserve. The development of high-resolution virtual slide scanners offers the possibility of quantifying, robustifying and accelerating the histopathological procedure. A major challenge for machine learning is to control the precision of predictions while enabling a high recall, in order to provide reproducibility. We use a multiple testing procedure that gives an overperforming way to solve the standard Precision-Recall trade-off that gives probabilistic guarantees on the precision. In addition, we significantly improve the overall performance of the models (increase of F1-score) by selecting the decision threshold using contextual biological information or using an auxiliary model. As it is model-agnostic, this contextual selection procedure paves the way to the development of a strategy that can improve the performance of any model without the need of retraining it. △ Less

Submitted 23 January, 2025; originally announced January 2025.

Comments: 11 pages, 3 figures

Cross--layer Formal Verification of Robotic Systems

Authors: Sylvain Raïs, Julien Brunel, David Doose, Frédéric Herbreteau

Abstract: Robotic systems are widely used to interact with humans or to perform critical tasks. As a result, it is imperative to provide guarantees about their behavior. Due to the modularity and complexity of robotic systems, their design and verification are often divided into several layers. However, some system properties can only be investigated by considering multiple layers simultaneously. We propose… ▽ More Robotic systems are widely used to interact with humans or to perform critical tasks. As a result, it is imperative to provide guarantees about their behavior. Due to the modularity and complexity of robotic systems, their design and verification are often divided into several layers. However, some system properties can only be investigated by considering multiple layers simultaneously. We propose a cross-layer verification method to verify the expected properties of concrete robotic systems. Our method verifies one layer using abstractions of other layers. We propose two approaches: refining the models of the abstract layers and refining the property under verification. A combination of these two approaches seems to be the most promising to ensure model genericity and to avoid the state-space explosion problem. △ Less

Submitted 21 November, 2024; originally announced November 2024.

Comments: In Proceedings FMAS2024, arXiv:2411.13215

Journal ref: EPTCS 411, 2024, pp. 143-150

Automatically Adaptive Conformal Risk Control

Authors: Vincent Blot, Anastasios N Angelopoulos, Michael I Jordan, Nicolas J-B Brunel

Abstract: Science and technology have a growing need for effective mechanisms that ensure reliable, controlled performance from black-box machine learning algorithms. These performance guarantees should ideally hold conditionally on the input-that is the performance guarantees should hold, at least approximately, no matter what the input. However, beyond stylized discrete groupings such as ethnicity and gen… ▽ More Science and technology have a growing need for effective mechanisms that ensure reliable, controlled performance from black-box machine learning algorithms. These performance guarantees should ideally hold conditionally on the input-that is the performance guarantees should hold, at least approximately, no matter what the input. However, beyond stylized discrete groupings such as ethnicity and gender, the right notion of conditioning can be difficult to define. For example, in problems such as image segmentation, we want the uncertainty to reflect the intrinsic difficulty of the test sample, but this may be difficult to capture via a conditioning event. Building on the recent work of Gibbs et al. [2023], we propose a methodology for achieving approximate conditional control of statistical risks-the expected value of loss functions-by adapting to the difficulty of test samples. Our framework goes beyond traditional conditional risk control based on user-provided conditioning events to the algorithmic, data-driven determination of appropriate function classes for conditioning. We apply this framework to various regression and segmentation tasks, enabling finer-grained control over model performance and demonstrating that by continuously monitoring and adjusting these parameters, we can achieve superior precision compared to conventional risk-control methods. △ Less

Submitted 27 March, 2025; v1 submitted 25 June, 2024; originally announced June 2024.

Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROA

Authors: Hussein Jawad, Yassine Chenik, Nicolas J. -B. Brunel

Abstract: The rapid adoption of Large Language Models (LLMs) has exposed critical security and ethical vulnerabilities, particularly their susceptibility to adversarial manipulations. This paper introduces QROA, a novel black-box jailbreak method designed to identify adversarial suffixes that can bypass LLM alignment safeguards when appended to a malicious instruction. Unlike existing suffix-based jailbreak… ▽ More The rapid adoption of Large Language Models (LLMs) has exposed critical security and ethical vulnerabilities, particularly their susceptibility to adversarial manipulations. This paper introduces QROA, a novel black-box jailbreak method designed to identify adversarial suffixes that can bypass LLM alignment safeguards when appended to a malicious instruction. Unlike existing suffix-based jailbreak approaches, QROA does not require access to the model's logit or any other internal information. It also eliminates reliance on human-crafted templates, operating solely through the standard query-response interface of LLMs. By framing the attack as an optimization bandit problem, QROA employs a surrogate model and token level optimization to efficiently explore suffix variations. Furthermore, we propose QROA-UNV, an extension that identifies universal adversarial suffixes for individual models, enabling one-query jailbreaks across a wide range of instructions. Testing on multiple models demonstrates Attack Success Rate (ASR) greater than 80\%. These findings highlight critical vulnerabilities, emphasize the need for advanced defenses, and contribute to the development of more robust safety evaluations for secure AI deployment. The code is made public on the following link: https://github.com/qroa/QROA △ Less

Submitted 6 May, 2025; v1 submitted 4 June, 2024; originally announced June 2024.

Adaptive Conformal Prediction by Reweighting Nonconformity Score

Authors: Salim I. Amoukou, Nicolas J. B Brunel

Abstract: Despite attractive theoretical guarantees and practical successes, Predictive Interval (PI) given by Conformal Prediction (CP) may not reflect the uncertainty of a given model. This limitation arises from CP methods using a constant correction for all test points, disregarding their individual uncertainties, to ensure coverage properties. To address this issue, we propose using a Quantile Regressi… ▽ More Despite attractive theoretical guarantees and practical successes, Predictive Interval (PI) given by Conformal Prediction (CP) may not reflect the uncertainty of a given model. This limitation arises from CP methods using a constant correction for all test points, disregarding their individual uncertainties, to ensure coverage properties. To address this issue, we propose using a Quantile Regression Forest (QRF) to learn the distribution of nonconformity scores and utilizing the QRF's weights to assign more importance to samples with residuals similar to the test point. This approach results in PI lengths that are more aligned with the model's uncertainty. In addition, the weights learnt by the QRF provide a partition of the features space, allowing for more efficient computations and improved adaptiveness of the PI through groupwise conformalization. Our approach enjoys an assumption-free finite sample marginal and training-conditional coverage, and under suitable assumptions, it also ensures conditional coverage. Our methods work for any nonconformity score and are available as a Python package. We conduct experiments on simulated and real-world data that demonstrate significant improvements compared to existing methods. △ Less

Submitted 31 May, 2023; v1 submitted 22 March, 2023; originally announced March 2023.

Local and Regional Counterfactual Rules: Summarized and Robust Recourses

Authors: Salim I. Amoukou, Nicolas J. B Brunel

Abstract: Counterfactual Explanations (CE) face several unresolved challenges, such as ensuring stability, synthesizing multiple CEs, and providing plausibility and sparsity guarantees. From a more practical point of view, recent studies [Pawelczyk et al., 2022] show that the prescribed counterfactual recourses are often not implemented exactly by individuals and demonstrate that most state-of-the-art CE al… ▽ More Counterfactual Explanations (CE) face several unresolved challenges, such as ensuring stability, synthesizing multiple CEs, and providing plausibility and sparsity guarantees. From a more practical point of view, recent studies [Pawelczyk et al., 2022] show that the prescribed counterfactual recourses are often not implemented exactly by individuals and demonstrate that most state-of-the-art CE algorithms are very likely to fail in this noisy environment. To address these issues, we propose a probabilistic framework that gives a sparse local counterfactual rule for each observation, providing rules that give a range of values capable of changing decisions with high probability. These rules serve as a summary of diverse counterfactual explanations and yield robust recourses. We further aggregate these local rules into a regional counterfactual rule, identifying shared recourses for subgroups of the data. Our local and regional rules are derived from the Random Forest algorithm, which offers statistical guarantees and fidelity to data distribution by selecting recourses in high-density regions. Moreover, our rules are sparse as we first select the smallest set of variables having a high probability of changing the decision. We have conducted experiments to validate the effectiveness of our counterfactual rules in comparison to standard CE and recent similar attempts. Our methods are available as a Python package. △ Less

Submitted 18 March, 2024; v1 submitted 29 September, 2022; originally announced September 2022.

Comments: ICML (International Conference on Machine Learning) 2023 Workshop on Counterfactuals in Minds and Machines

Journal ref: ICML (International Conference on Machine Learning) 2023 Workshop on Counterfactuals in Minds and Machines

Consistent Sufficient Explanations and Minimal Local Rules for explaining regression and classification models

Authors: Salim I. Amoukou, Nicolas J. B Brunel

Abstract: To explain the decision of any model, we extend the notion of probabilistic Sufficient Explanations (P-SE). For each instance, this approach selects the minimal subset of features that is sufficient to yield the same prediction with high probability, while removing other features. The crux of P-SE is to compute the conditional probability of maintaining the same prediction. Therefore, we introduce… ▽ More To explain the decision of any model, we extend the notion of probabilistic Sufficient Explanations (P-SE). For each instance, this approach selects the minimal subset of features that is sufficient to yield the same prediction with high probability, while removing other features. The crux of P-SE is to compute the conditional probability of maintaining the same prediction. Therefore, we introduce an accurate and fast estimator of this probability via random Forests for any data $(\boldsymbol{X}, Y)$ and show its efficiency through a theoretical analysis of its consistency. As a consequence, we extend the P-SE to regression problems. In addition, we deal with non-discrete features, without learning the distribution of $\boldsymbol{X}$ nor having the model for making predictions. Finally, we introduce local rule-based explanations for regression/classification based on the P-SE and compare our approaches w.r.t other explainable AI methods. These methods are available as a Python package at \url{www.github.com/salimamoukou/acv00}. △ Less

Submitted 14 October, 2022; v1 submitted 8 November, 2021; originally announced November 2021.

Comments: Accepted at the 36th Conference on Neural Information Processing Systems (NeurIPS 2022)

Accurate Shapley Values for explaining tree-based models

Authors: Salim I. Amoukou, Nicolas J-B. Brunel, Tangi Salaün

Abstract: Shapley Values (SV) are widely used in explainable AI, but their estimation and interpretation can be challenging, leading to inaccurate inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we int… ▽ More Shapley Values (SV) are widely used in explainable AI, but their estimation and interpretation can be challenging, leading to inaccurate inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we introduce two estimators of Shapley Values that exploit the tree structure efficiently and are more accurate than state-of-the-art methods. Simulations and comparisons are performed with state-of-the-art algorithms and show the practical gain of our approach. Finally, we discuss the limitations of Shapley Values as a local explanation. These methods are available as a Python package. △ Less

Submitted 31 May, 2023; v1 submitted 7 June, 2021; originally announced June 2021.

Comments: Accepted at the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022. V2: The section on Active Shapley Values has been removed in this updated version

Journal ref: AISTATS 2022

The Shapley Value of coalition of variables provides better explanations

Authors: Salim I. Amoukou, Nicolas J-B. Brunel, Tangi Salaün

Abstract: While Shapley Values (SV) are one of the gold standard for interpreting machine learning models, we show that they are still poorly understood, in particular in the presence of categorical variables or of variables of low importance. For instance, we show that the popular practice that consists in summing the SV of dummy variables is false as it provides wrong estimates of all the SV in the model… ▽ More While Shapley Values (SV) are one of the gold standard for interpreting machine learning models, we show that they are still poorly understood, in particular in the presence of categorical variables or of variables of low importance. For instance, we show that the popular practice that consists in summing the SV of dummy variables is false as it provides wrong estimates of all the SV in the model and implies spurious interpretations. Based on the identification of null and active coalitions, and a coalitional version of the SV, we provide a correct computation and inference of important variables. Moreover, a Python library (All the experiments and simulations can be reproduced with the publicly available library Active Coalition of Variables, https://www.github.com/salimamoukou/acv00) that computes reliably conditional expectations and SV for tree-based models, is implemented and compared with state-of-the-art algorithms on toy models and real data sets. △ Less

Submitted 6 April, 2022; v1 submitted 24 March, 2021; originally announced March 2021.

Comments: This paper has been withdrawn by the authors, because it has now been merged with (and superseded by) a parallel work arXiv:2106.03820

Simulation under Arbitrary Temporal Logic Constraints

Authors: Julien Brunel, David Chemouil, Alcino Cunha, Nuno Macedo

Abstract: Most model checkers provide a useful simulation mode, that allows users to explore the set of possible behaviours by interactively picking at each state which event to execute next. Traditionally this simulation mode cannot take into consideration additional temporal logic constraints, such as arbitrary fairness restrictions, substantially reducing its usability for debugging the modelled system b… ▽ More Most model checkers provide a useful simulation mode, that allows users to explore the set of possible behaviours by interactively picking at each state which event to execute next. Traditionally this simulation mode cannot take into consideration additional temporal logic constraints, such as arbitrary fairness restrictions, substantially reducing its usability for debugging the modelled system behaviour. Similarly, when a specification is false, even if all its counter-examples combined also form a set of behaviours, most model checkers only present one of them to the user, providing little or no mechanism to explore alternatives. In this paper, we present a simple on-the-fly verification technique to allow the user to explore the behaviours that satisfy an arbitrary temporal logic specification, with an interactive process akin to simulation. This technique enables a unified interface for simulating the modelled system and exploring its counter-examples. The technique is formalised in the framework of state/event linear temporal logic and a proof of concept was implemented in an event-based variant of the Electrum framework. △ Less

Submitted 23 December, 2019; originally announced December 2019.

Comments: In Proceedings F-IDE 2019, arXiv:1912.09611

Journal ref: EPTCS 310, 2019, pp. 63-69

Formal Safety and Security Assessment of an Avionic Architecture with Alloy

Authors: Julien Brunel, Laurent Rioux, Stéphane Paul, Anthony Faucogney, Frédérique Vallée

Abstract: We propose an approach based on Alloy to formally model and assess a system architecture with respect to safety and security requirements. We illustrate this approach by considering as a case study an avionic system developed by Thales, which provides guidance to aircraft. We show how to define in Alloy a metamodel of avionic architectures with a focus on failure propagations. We then express… ▽ More We propose an approach based on Alloy to formally model and assess a system architecture with respect to safety and security requirements. We illustrate this approach by considering as a case study an avionic system developed by Thales, which provides guidance to aircraft. We show how to define in Alloy a metamodel of avionic architectures with a focus on failure propagations. We then express the specific architecture of the case study in Alloy. Finally, we express and check properties that refer to the robustness of the architecture to failures and attacks. △ Less

Submitted 5 May, 2014; originally announced May 2014.

Comments: In Proceedings ESSS 2014, arXiv:1405.0554

Journal ref: EPTCS 150, 2014, pp. 8-19

Towards an Updatable Strategy Logic

Authors: Christophe Chareton, Julien Brunel, David Chemouil

Abstract: This article is about temporal multi-agent logics. Several of these formalisms have been already presented (ATL-ATL*, ATLsc, SL). They enable to express the capacities of agents in a system to ensure the satisfaction of temporal properties. Particularly, SL and ATLsc enable several agents to interact in a context mixing the different strategies they play in a semantical game. We generalize this po… ▽ More This article is about temporal multi-agent logics. Several of these formalisms have been already presented (ATL-ATL*, ATLsc, SL). They enable to express the capacities of agents in a system to ensure the satisfaction of temporal properties. Particularly, SL and ATLsc enable several agents to interact in a context mixing the different strategies they play in a semantical game. We generalize this possibility by proposing a new formalism, Updating Strategy Logic (USL). In USL, an agent can also refine its own strategy. The gain in expressive power rises the notion of "sustainable capacities" for agents. USL is built from SL. It mainly brings to SL the two following modifications: semantically, the successor of a given state is not uniquely determined by the data of one choice from each agent. Syntactically, we introduce in the language an operator, called an "unbinder", which explicitely deletes the binding of a strategy to an agent. We show that USL is strictly more expressive than SL. △ Less

Submitted 4 March, 2013; originally announced March 2013.

Comments: In Proceedings SR 2013, arXiv:1303.0071

Journal ref: EPTCS 112, 2013, pp. 91-98