Showing 1–19 of 19 results for author: de Carvalho, M

A Kolmogorov-Arnold Neural Model for Cascading Extremes

Authors: Miguel de Carvalho, Clemente Ferrer, Ronny Vallejos

Abstract: This paper addresses the growing concern of cascading extreme events, such as an extreme earthquake followed by a tsunami, by presenting a novel method for risk assessment focused on these domino effects. The proposed approach develops an extreme value theory framework within a Kolmogorov-Arnold network (KAN) to estimate the probability of one extreme event triggering another, conditionally on a f… ▽ More This paper addresses the growing concern of cascading extreme events, such as an extreme earthquake followed by a tsunami, by presenting a novel method for risk assessment focused on these domino effects. The proposed approach develops an extreme value theory framework within a Kolmogorov-Arnold network (KAN) to estimate the probability of one extreme event triggering another, conditionally on a feature vector. An extra layer is added to the KAN's architecture to enforce the definition of the parameter of interest within the unit interval, and we refer to the resulting neural model as KANE (KAN with Natural Enforcement). The proposed method is backed by exhaustive numerical studies and further illustrated with real-world applications to seismology and climatology. △ Less

Submitted 19 May, 2025; originally announced May 2025.

When a Reinforcement Learning Agent Encounters Unknown Unknowns

Authors: Juntian Zhu, Miguel de Carvalho, Zhouwang Yang, Fengxiang He

Abstract: An AI agent might surprisingly find she has reached an unknown state which she has never been aware of -- an unknown unknown. We mathematically ground this scenario in reinforcement learning: an agent, after taking an action calculated from value functions $Q$ and $V$ defined on the {\it {aware domain}}, reaches a state out of the domain. To enable the agent to handle this scenario, we propose an… ▽ More An AI agent might surprisingly find she has reached an unknown state which she has never been aware of -- an unknown unknown. We mathematically ground this scenario in reinforcement learning: an agent, after taking an action calculated from value functions $Q$ and $V$ defined on the {\it {aware domain}}, reaches a state out of the domain. To enable the agent to handle this scenario, we propose an {\it episodic Markov decision {process} with growing awareness} (EMDP-GA) model, taking a new {\it noninformative value expansion} (NIVE) approach to expand value functions to newly aware areas: when an agent arrives at an unknown unknown, value functions $Q$ and $V$ whereon are initialised by noninformative beliefs -- the averaged values on the aware domain. This design is out of respect for the complete absence of knowledge in the newly discovered state. The upper confidence bound momentum Q-learning is then adapted to the growing awareness for training the EMDP-GA model. We prove that (1) the regret of our approach is asymptotically consistent with the state of the art (SOTA) without exposure to unknown unknowns in an extremely uncertain environment, and (2) our computational complexity and space complexity are comparable with the SOTA -- these collectively suggest that though an unknown unknown is surprising, it will be asymptotically properly discovered with decent speed and an affordable cost. △ Less

Submitted 19 May, 2025; originally announced May 2025.

The underlap coefficient as a measure of a biomarker's discriminatory ability

Authors: Zhaoxi Zhang, Vanda Inacio, Miguel de Carvalho

Abstract: The first step in evaluating a potential diagnostic biomarker is to examine the variation in its values across different disease groups. In a three-class disease setting, the volume under the receiver operating characteristic surface and the three-class Youden index are commonly used summary measures of a biomarker's discriminatory ability. However, these measures rely on a stochastic ordering ass… ▽ More The first step in evaluating a potential diagnostic biomarker is to examine the variation in its values across different disease groups. In a three-class disease setting, the volume under the receiver operating characteristic surface and the three-class Youden index are commonly used summary measures of a biomarker's discriminatory ability. However, these measures rely on a stochastic ordering assumption for the distributions of biomarker outcomes across the three groups. This assumption can be restrictive, particularly when covariates are involved, and its violation may lead to incorrect conclusions about a biomarker's ability to distinguish between the three disease classes. Even when a stochastic ordering exists, the order may vary across different biomarkers in discovery studies involving dozens or even thousands of candidate biomarkers, complicating automated ranking. To address these challenges and complement existing measures, we propose the underlap coefficient, a novel summary index of a biomarker's ability to distinguish between three (or more) disease groups, and study its properties. Additionally, we introduce Bayesian nonparametric estimators for both the unconditional underlap coefficient and its covariate-specific counterpart. These estimators are broadly applicable to a wide range of biomarkers and populations. A simulation study reveals a good performance of the proposed estimators across a range of conceivable scenarios. We illustrate the proposed approach through an application to an Alzheimer's disease (AD) dataset aimed to assess how four potential AD biomarkers distinguish between individuals with normal cognition, mild impairment, and dementia, and how and if age and gender impact this discriminatory ability. △ Less

Submitted 17 April, 2025; v1 submitted 16 April, 2025; originally announced April 2025.

Decoding AI: The inside story of data analysis in ChatGPT

Authors: Ozan Evkaya, Miguel de Carvalho

Abstract: As a result of recent advancements in generative AI, the field of Data Science is prone to various changes. This review critically examines the Data Analysis (DA) capabilities of ChatGPT assessing its performance across a wide range of tasks. While DA provides researchers and practitioners with unprecedented analytical capabilities, it is far from being perfect, and it is important to recognize an… ▽ More As a result of recent advancements in generative AI, the field of Data Science is prone to various changes. This review critically examines the Data Analysis (DA) capabilities of ChatGPT assessing its performance across a wide range of tasks. While DA provides researchers and practitioners with unprecedented analytical capabilities, it is far from being perfect, and it is important to recognize and address its limitations. △ Less

Submitted 12 April, 2024; originally announced April 2024.

Comments: 15 pages with figures and appendix

A parallelizable model-based approach for marginal and multivariate clustering

Authors: Miguel de Carvalho, Gabriel Martos Venturini, Andrej Svetlošák

Abstract: This paper develops a clustering method that takes advantage of the sturdiness of model-based clustering, while attempting to mitigate some of its pitfalls. First, we note that standard model-based clustering likely leads to the same number of clusters per margin, which seems a rather artificial assumption for a variety of datasets. We tackle this issue by specifying a finite mixture model per mar… ▽ More This paper develops a clustering method that takes advantage of the sturdiness of model-based clustering, while attempting to mitigate some of its pitfalls. First, we note that standard model-based clustering likely leads to the same number of clusters per margin, which seems a rather artificial assumption for a variety of datasets. We tackle this issue by specifying a finite mixture model per margin that allows each margin to have a different number of clusters, and then cluster the multivariate data using a strategy game-inspired algorithm to which we call Reign-and-Conquer. Second, since the proposed clustering approach only specifies a model for the margins -- but leaves the joint unspecified -- it has the advantage of being partially parallelizable; hence, the proposed approach is computationally appealing as well as more tractable for moderate to high dimensions than a `full' (joint) model-based clustering approach. A battery of numerical experiments on artificial data indicate an overall good performance of the proposed methods in a variety of scenarios, and real datasets are used to showcase their application in practice. △ Less

Submitted 7 December, 2022; originally announced December 2022.

Uncovering Regions of Maximum Dissimilarity on Random Process Data

Authors: Miguel de Carvalho, Gabriel Martos Venturini

Abstract: The comparison of local characteristics of two random processes can shed light on periods of time or space at which the processes differ the most. This paper proposes a method that learns about regions with a certain volume, where the marginal attributes of two processes are less similar. The proposed methods are devised in full generality for the setting where the data of interest are themselves… ▽ More The comparison of local characteristics of two random processes can shed light on periods of time or space at which the processes differ the most. This paper proposes a method that learns about regions with a certain volume, where the marginal attributes of two processes are less similar. The proposed methods are devised in full generality for the setting where the data of interest are themselves stochastic processes, and thus the proposed method can be used for pointing out the regions of maximum dissimilarity with a certain volume, in the contexts of functional data, time series, and point processes. The parameter functions underlying both stochastic processes of interest are modeled via a basis representation, and Bayesian inference is conducted via an integrated nested Laplace approximation. The numerical studies validate the proposed methods, and we showcase their application with case studies on criminology, finance, and medicine. △ Less

Submitted 12 September, 2022; originally announced September 2022.

ACM Class: G.3

Tracking change-points in multivariate extremes

Authors: Miguel de Carvalho, Manuele Leonelli, Alex Rossi

Abstract: In this paper we devise a statistical method for tracking and modeling change-points on the dependence structure of multivariate extremes. The methods are motivated by and illustrated on a case study on crypto-assets. In this paper we devise a statistical method for tracking and modeling change-points on the dependence structure of multivariate extremes. The methods are motivated by and illustrated on a case study on crypto-assets. △ Less

Submitted 10 November, 2020; originally announced November 2020.

Modeling Interval Trendlines: Symbolic Singular Spectrum Analysis for Interval Time Series

Authors: Miguel de Carvalho, Gabriel Martos

Abstract: In this article we propose an extension of singular spectrum analysis for interval-valued time series. The proposed methods can be used to decompose and forecast the dynamics governing a set-valued stochastic process. The resulting components on which the interval time series is decomposed can be understood as interval trendlines, cycles, or noise. Forecasting can be conducted through a linear rec… ▽ More In this article we propose an extension of singular spectrum analysis for interval-valued time series. The proposed methods can be used to decompose and forecast the dynamics governing a set-valued stochastic process. The resulting components on which the interval time series is decomposed can be understood as interval trendlines, cycles, or noise. Forecasting can be conducted through a linear recurrent method, and we devised generalizations of the decomposition method for the multivariate setting. The performance of the proposed methods is showcased in a simulation study. We apply the proposed methods so to track the dynamics governing the Argentina Stock Market (MERVAL) in real time, in a case study that covers the most recent period of turbulence that led to discussions of the government of Argentina with the International Monetary Fund. △ Less

Submitted 7 November, 2020; originally announced November 2020.

An Extreme Value Bayesian Lasso for the Conditional Left and Right Tails

Authors: Miguel de Carvalho, Soraia Pereira, Paula Pereira, Patrícia de Zea Bermudez

Abstract: We introduce a novel regression model for the conditional left and right tail of a possibly heavy-tailed response. The proposed model can be used to learn the effect of covariates on an extreme value setting via a Lasso-type specification based on a Lagrangian restriction. Our model can be used to track if some covariates are significant for the lower values, but not for the (right) tail---and vic… ▽ More We introduce a novel regression model for the conditional left and right tail of a possibly heavy-tailed response. The proposed model can be used to learn the effect of covariates on an extreme value setting via a Lasso-type specification based on a Lagrangian restriction. Our model can be used to track if some covariates are significant for the lower values, but not for the (right) tail---and vice-versa; in addition to this, the proposed model bypasses the need for conditional threshold selection in an extreme value theory framework. We assess the finite-sample performance of the proposed methods through a simulation study that reveals that our method recovers the true conditional distribution over a variety of simulation scenarios, along with being accurate on variable selection. Rainfall data are used to showcase how the proposed method can learn to distinguish between key drivers of moderate rainfall, against those of extreme rainfall. △ Less

Submitted 10 August, 2021; v1 submitted 14 October, 2020; originally announced October 2020.

Robust and flexible inference for the covariate-specific ROC curve

Authors: Vanda Inacio, Vanda M. Lourenco, Miguel de Carvalho, Richard A. Parker, Vincent Gnanapragasam

Abstract: Diagnostic tests are of critical importance in health care and medical research. Motivated by the impact that atypical and outlying test outcomes might have on the assessment of the discriminatory ability of a diagnostic test, we develop a flexible and robust model for conducting inference about the covariate-specific receiver operating characteristic (ROC) curve that safeguards against outlying t… ▽ More Diagnostic tests are of critical importance in health care and medical research. Motivated by the impact that atypical and outlying test outcomes might have on the assessment of the discriminatory ability of a diagnostic test, we develop a flexible and robust model for conducting inference about the covariate-specific receiver operating characteristic (ROC) curve that safeguards against outlying test results while also accommodating for possible nonlinear effects of the covariates. Specifically, we postulate a location-scale additive regression model for the test outcomes in both the diseased and nondiseased populations, combining additive cubic B-splines and M-estimation for the regression function, while the residuals are estimated via a weighted empirical distribution function. The results of the simulation study show that our approach successfully recovers the true covariate-specific ROC curve and corresponding area under the curve on a variety of conceivable test outcomes contamination scenarios. Our method is applied to a dataset derived from a prostate cancer study where we seek to assess the ability of the Prostate Health Index to discriminate between men with and without Gleason 7 or above prostate cancer, and if and how such discriminatory capacity changes with age. △ Less

Submitted 27 July, 2020; v1 submitted 12 July, 2020; originally announced July 2020.

ATL: Autonomous Knowledge Transfer from Many Streaming Processes

Authors: Mahardhika Pratama, Marcus de Carvalho, Renchunzi Xie, Edwin Lughofer, Jie Lu

Abstract: Transferring knowledge across many streaming processes remains an uncharted territory in the existing literature and features unique characteristics: no labelled instance of the target domain, covariate shift of source and target domain, different period of drifts in the source and target domains. Autonomous transfer learning (ATL) is proposed in this paper as a flexible deep learning approach for… ▽ More Transferring knowledge across many streaming processes remains an uncharted territory in the existing literature and features unique characteristics: no labelled instance of the target domain, covariate shift of source and target domain, different period of drifts in the source and target domains. Autonomous transfer learning (ATL) is proposed in this paper as a flexible deep learning approach for the online unsupervised transfer learning problem across many streaming processes. ATL offers an online domain adaptation strategy via the generative and discriminative phases coupled with the KL divergence based optimization strategy to produce a domain invariant network while putting forward an elastic network structure. It automatically evolves its network structure from scratch with/without the presence of ground truth to overcome independent concept drifts in the source and target domain. The rigorous numerical evaluation has been conducted along with a comparison against recently published works. ATL demonstrates improved performance while showing significantly faster training speed than its counterparts. △ Less

Submitted 19 October, 2019; v1 submitted 8 October, 2019; originally announced October 2019.

Comments: This paper has been accepted for publication in CIKM 2019

Predicting assisted ventilation in Amyotrophic Lateral Sclerosis using a mixture of experts and conformal predictors

Authors: Telma Pereira, Sofia Pires, Marta Gromicho, Susana Pinto, Mamede de Carvalho, Sara C. Madeira

Abstract: Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease characterized by a rapid motor decline, leading to respiratory failure and subsequently to death. In this context, researchers have sought for models to automatically predict disease progression to assisted ventilation in ALS patients. However, the clinical translation of such models is limited by the lack of insight 1) on the risk… ▽ More Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease characterized by a rapid motor decline, leading to respiratory failure and subsequently to death. In this context, researchers have sought for models to automatically predict disease progression to assisted ventilation in ALS patients. However, the clinical translation of such models is limited by the lack of insight 1) on the risk of error for predictions at patient-level, and 2) on the most adequate time to administer the non-invasive ventilation. To address these issues, we combine Conformal Prediction (a machine learning framework that complements predictions with confidence measures) and a mixture experts into a prognostic model which not only predicts whether an ALS patient will suffer from respiratory insufficiency but also the most likely time window of occurrence, at a given reliability level. Promising results were obtained, with near 80% of predictions being correctly identified. △ Less

Submitted 30 July, 2019; originally announced July 2019.

Journal ref: KDD 2019 Workshop on Applied Data Science for Healthcare

Bayesian semiparametric modelling of phase-varying point processes

Authors: Bastian Galasso, Yoav Zemel, Miguel de Carvalho

Abstract: We propose a Bayesian semiparametric approach for registration of multiple point processes. Our approach entails modelling the mean measures of the phase-varying point processes with a Bernstein-Dirichlet prior, which induces a prior on the space of all warp functions. Theoretical results on the support of the induced priors are derived, and posterior consistency is obtained under mild conditions.… ▽ More We propose a Bayesian semiparametric approach for registration of multiple point processes. Our approach entails modelling the mean measures of the phase-varying point processes with a Bernstein-Dirichlet prior, which induces a prior on the space of all warp functions. Theoretical results on the support of the induced priors are derived, and posterior consistency is obtained under mild conditions. Numerical experiments suggest a good performance of the proposed methods, and a climatology real-data example is used to showcase how the method can be employed in practice. △ Less

Submitted 11 December, 2020; v1 submitted 22 December, 2018; originally announced December 2018.

Comments: 30 pages, 16 figures

MSC Class: 60G55; 62G99; 62F15

Bayesian Bootstrap Inference for the ROC Surface

Authors: Vanda Inacio de Carvalho, Miguel de Carvalho, Adam Branscum

Abstract: Accurate diagnosis of disease is of great importance in clinical practice and medical research. The receiver operating characteristic (ROC) surface is a popular tool for evaluating the discriminatory ability of continuous diagnostic test outcomes when there exist three ordered disease classes (e.g., no disease, mild disease, advanced disease). We propose the Bayesian bootstrap, a fully nonparametr… ▽ More Accurate diagnosis of disease is of great importance in clinical practice and medical research. The receiver operating characteristic (ROC) surface is a popular tool for evaluating the discriminatory ability of continuous diagnostic test outcomes when there exist three ordered disease classes (e.g., no disease, mild disease, advanced disease). We propose the Bayesian bootstrap, a fully nonparametric method, for conducting inference about the ROC surface and its functionals, such as the volume under the surface. The proposed method is based on a simple, yet interesting, representation of the ROC surface in terms of placement variables. Results from a simulation study demonstrate the ability of our method to successfully recover the true ROC surface and to produce valid inferences in a variety of complex scenarios. An application to data from the Trail Making Test to assess cognitive impairment in Parkinson's disease patients is provided. △ Less

Submitted 19 May, 2018; originally announced May 2018.

Affinity-based measures of medical diagnostic test accuracy

Authors: Miguel de Carvalho, Bradley J. Barney, Garritt L. Page

Abstract: We propose new summary measures of diagnostic test accuracy which can be used as companions to existing diagnostic accuracy measures. Conceptually, our summary measures are tantamount to the so-called Hellinger affinity and we show that they can be regarded as measures of agreement constructed from similar geometrical principles as Pearson correlation. A covariate-specific version of our summary i… ▽ More We propose new summary measures of diagnostic test accuracy which can be used as companions to existing diagnostic accuracy measures. Conceptually, our summary measures are tantamount to the so-called Hellinger affinity and we show that they can be regarded as measures of agreement constructed from similar geometrical principles as Pearson correlation. A covariate-specific version of our summary index is developed, which can be used to assess the discrimination performance of a diagnostic test, conditionally on the value of a predictor. Nonparametric Bayes estimators for the proposed indexes are devised, theoretical properties of the corresponding priors are derived, and the performance of our methods is assessed through a simulation study. Data from a prostate cancer diagnosis study are used to illustrate our methods. △ Less

Submitted 28 December, 2017; originally announced December 2017.

Regression Type Models for Extremal Dependence

Authors: Linda Mhalla, Miguel de Carvalho, Valérie Chavez-Demoulin

Abstract: We propose a vector generalized additive modeling framework for taking into account the effect of covariates on angular density functions in a multivariate extreme value context. The proposed methods are tailored for settings where the dependence between extreme values may change according to covariates. We devise a maximum penalized log-likelihood estimator, discuss details of the estimation proc… ▽ More We propose a vector generalized additive modeling framework for taking into account the effect of covariates on angular density functions in a multivariate extreme value context. The proposed methods are tailored for settings where the dependence between extreme values may change according to covariates. We devise a maximum penalized log-likelihood estimator, discuss details of the estimation procedure, and derive its consistency and asymptotic normality. The simulation study suggests that the proposed methods perform well in a wealth of simulation scenarios by accurately recovering the true covariate-adjusted angular density. Our empirical analysis reveals relevant dynamics of the dependence between extreme air temperatures in two alpine resorts during the winter season. Supplementary materials for this article are available online. △ Less

Submitted 27 November, 2017; v1 submitted 27 April, 2017; originally announced April 2017.

Comments: 29 pages, 8 figures

On the geometry of Bayesian inference

Authors: Miguel de Carvalho, Garritt L. Page, Bradley J. Barney

Abstract: We provide a geometric interpretation to Bayesian inference that allows us to introduce a natural measure of the level of agreement between priors, likelihoods, and posteriors. The starting point for the construction of our geometry is the simple observation that the marginal likelihood can be regarded as an inner product between the prior and the likelihood. A key concept in our geometry is that… ▽ More We provide a geometric interpretation to Bayesian inference that allows us to introduce a natural measure of the level of agreement between priors, likelihoods, and posteriors. The starting point for the construction of our geometry is the simple observation that the marginal likelihood can be regarded as an inner product between the prior and the likelihood. A key concept in our geometry is that of compatibility, a measure which is based on the same construction principles as Pearson correlation, but which can be used to assess how much the prior agrees with the likelihood, to gauge the sensitivity of the posterior to the prior, and to quantify the coherency of the opinions of two experts. Estimators for all the quantities involved in our geometric setup are discussed, which can be directly computed from the posterior simulation output. Some examples are used to illustrate our methods, including data related to on-the-job drug usage, midge wing length, and prostate cancer. △ Less

Submitted 23 May, 2018; v1 submitted 31 January, 2017; originally announced January 2017.

Combining probability distributions: Extending the logarithmic pooling approach

Authors: Luiz Max de Carvalho, Daniel A. M. Villela, Flavio Codeco Coelho, Leonardo Soares Bastos

Abstract: Combining distributions is an important issue in decision theory and Bayesian inference. Logarithmic pooling is a popular method to aggregate expert opinions by using a set of weights that reflect the reliability of each information source. However, the resulting pooled distribution depends heavily on set of weights given to each opinion/prior and thus careful consideration must be given to the ch… ▽ More Combining distributions is an important issue in decision theory and Bayesian inference. Logarithmic pooling is a popular method to aggregate expert opinions by using a set of weights that reflect the reliability of each information source. However, the resulting pooled distribution depends heavily on set of weights given to each opinion/prior and thus careful consideration must be given to the choice of weights. In this paper we review and extend the statistical theory of logarithmic pooling, focusing on the assignment of the weights using a hierarchical prior distribution. We explore several statistical applications, such as the estimation of survival probabilities, meta-analysis and Bayesian melding of deterministic models of population growth and epidemics. We show that it is possible learn the weights from data, although identifiability issues may arise for some configurations of priors and data. Furthermore, we show how the hierarchical approach leads to posterior distributions that are able to accommodate prior-data conflict in complex models. △ Less

Submitted 30 December, 2020; v1 submitted 14 February, 2015; originally announced February 2015.

Comments: Massively updated manuscript; submitted for publication

A Euclidean likelihood estimator for bivariate tail dependence

Authors: Miguel de Carvalho, Boris Oumow, Johan Segers, Michał Warchoł

Abstract: The spectral measure plays a key role in the statistical modeling of multivariate extremes. Estimation of the spectral measure is a complex issue, given the need to obey a certain moment condition. We propose a Euclidean likelihood-based estimator for the spectral measure which is simple and explicitly defined, with its expression being free of Lagrange multipliers. Our estimator is shown to have… ▽ More The spectral measure plays a key role in the statistical modeling of multivariate extremes. Estimation of the spectral measure is a complex issue, given the need to obey a certain moment condition. We propose a Euclidean likelihood-based estimator for the spectral measure which is simple and explicitly defined, with its expression being free of Lagrange multipliers. Our estimator is shown to have the same limit distribution as the maximum empirical likelihood estimator of J. H. J. Einmahl and J. Segers, Annals of Statistics 37(5B), 2953--2989 (2009). Numerical experiments suggest an overall good performance and identical behavior to the maximum empirical likelihood estimator. We illustrate the method in an extreme temperature data analysis. △ Less

Submitted 16 April, 2012; originally announced April 2012.

Comments: 18 pages, 8 figures

MSC Class: 62G32