-
Using permutations to assess confounding in machine learning applications for digital health
Authors:
Elias Chaibub Neto,
Abhishek Pratap,
Thanneer M Perumal,
Meghasyam Tummalacherla,
Brian M Bot,
Lara Mangravite,
Larsson Omberg
Abstract:
Clinical machine learning applications are often plagued with confounders that can impact the generalizability and predictive performance of the learners. Confounding is especially problematic in remote digital health studies where the participants self-select to enter the study, thereby making it challenging to balance the demographic characteristics of participants. One effective approach to com…
▽ More
Clinical machine learning applications are often plagued with confounders that can impact the generalizability and predictive performance of the learners. Confounding is especially problematic in remote digital health studies where the participants self-select to enter the study, thereby making it challenging to balance the demographic characteristics of participants. One effective approach to combat confounding is to match samples with respect to the confounding variables in order to balance the data. This procedure, however, leads to smaller datasets and hence impact the inferences drawn from the learners. Alternatively, confounding adjustment methods that make more efficient use of the data (e.g., inverse probability weighting) usually rely on modeling assumptions, and it is unclear how robust these methods are to violations of these assumptions. Here, rather than proposing a new approach to control for confounding, we develop novel permutation based statistical methods to detect and quantify the influence of observed confounders, and estimate the unconfounded performance of the learner. Our tools can be used to evaluate the effectiveness of existing confounding adjustment methods. We illustrate their application using real-life data from a Parkinson's disease mobile health study collected in an uncontrolled environment.
△ Less
Submitted 28 November, 2018;
originally announced November 2018.
-
Learning Disease vs Participant Signatures: a permutation test approach to detect identity confounding in machine learning diagnostic applications
Authors:
Elias Chaibub Neto,
Abhishek Pratap,
Thanneer M Perumal,
Meghasyam Tummalacherla,
Brian M Bot,
Andrew D Trister,
Stephen H Friend,
Lara Mangravite,
Larsson Omberg
Abstract:
Recently, Saeb et al (2017) showed that, in diagnostic machine learning applications, having data of each subject randomly assigned to both training and test sets (record-wise data split) can lead to massive underestimation of the cross-validation prediction error, due to the presence of "subject identity confounding" caused by the classifier's ability to identify subjects, instead of recognizing…
▽ More
Recently, Saeb et al (2017) showed that, in diagnostic machine learning applications, having data of each subject randomly assigned to both training and test sets (record-wise data split) can lead to massive underestimation of the cross-validation prediction error, due to the presence of "subject identity confounding" caused by the classifier's ability to identify subjects, instead of recognizing disease. To solve this problem, the authors recommended the random assignment of the data of each subject to either the training or the test set (subject-wise data split). The adoption of subject-wise split has been criticized in Little et al (2017), on the basis that it can violate assumptions required by cross-validation to consistently estimate generalization error. In particular, adopting subject-wise splitting in heterogeneous data-sets might lead to model under-fitting and larger classification errors. Hence, Little et al argue that perhaps the overestimation of prediction errors with subject-wise cross-validation, rather than underestimation with record-wise cross-validation, is the reason for the discrepancies between prediction error estimates generated by the two splitting strategies. In order to shed light on this controversy, we focus on simpler classification performance metrics and develop permutation tests that can detect identity confounding. By focusing on permutation tests, we are able to evaluate the merits of record-wise and subject-wise data splits under more general statistical dependencies and distributional structures of the data, including situations where cross-validation breaks down. We illustrate the application of our tests using synthetic and real data from a Parkinson's disease study.
△ Less
Submitted 6 July, 2018; v1 submitted 8 December, 2017;
originally announced December 2017.
-
On the analysis of personalized medication response and classification of case vs control patients in mobile health studies: the mPower case study
Authors:
Elias Chaibub Neto,
Thanneer M Perumal,
Abhishek Pratap,
Brian M Bot,
Lara Mangravite,
Larsson Omberg
Abstract:
In this work we provide a couple of contributions to the analysis of longitudinal data collected by smartphones in mobile health applications. First, we propose a novel statistical approach to disentangle personalized treatment and "time-of-the-day" effects in observational studies. Under the assumption of no unmeasured confounders, we show how to use conditional independence relations in the data…
▽ More
In this work we provide a couple of contributions to the analysis of longitudinal data collected by smartphones in mobile health applications. First, we propose a novel statistical approach to disentangle personalized treatment and "time-of-the-day" effects in observational studies. Under the assumption of no unmeasured confounders, we show how to use conditional independence relations in the data in order to determine if a difference in performance between activity tasks performed before and after the participant has taken medication, are potentially due to an effect of the medication or to a "time-of-the-day" effect (or still to both). Second, we show that smartphone data collected from a given study participant can represent a "digital fingerprint" of the participant, and that classifiers of case/control labels, constructed using longitudinal data, can show artificially improved performance when data from each participant is included in both training and test sets. We illustrate our contributions using data collected during the first 6 months of the mPower study.
△ Less
Submitted 29 June, 2017;
originally announced June 2017.
