Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
Authors:
Usman Mahmood,
Robik Shrestha,
David D. B. Bates,
Lorenzo Mannelli,
Giuseppe Corrias,
Yusuf Erdi,
Christopher Kanan
Abstract:
Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safe…
▽ More
Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safety. The current gold standard approach is to conduct an analytical validation of performance on a generalization dataset from one or more institutions, followed by a clinical validation study of the system's efficacy during deployment. Clinical validation studies are time-consuming, and best practices dictate limited re-use of analytical validation data, so it is ideal to know ahead of time if a system is likely to fail analytical or clinical validation. In this paper, we describe a series of sanity tests to identify when a system performs well on development data for the wrong reasons. We illustrate the sanity tests' value by designing a deep learning system to classify pancreatic cancer seen in computed tomography scans.
△ Less
Submitted 4 March, 2021;
originally announced March 2021.
Citizen COmputing for Pulsar Searches: CICLOPS
Authors:
Matteo Bachetti,
Maura Pilia,
Stefano Curatti,
Giada Corrias,
Andrea Addis,
Claudia Macciò,
Daniele Muntoni,
Viviana Piga,
Nicolò Pitzalis,
Alessio Trois
Abstract:
Most periodicity search algorithms used in pulsar astronomy today are highly efficient and take advantage of multiple CPUs or GPUs. The bottlenecks are usually represented by the operations that require an informed choice from an expert eye. A typical case is the presence of radio-frequency interferences in the data, that often mimic the periodic signals of pulsars, and require visual inspection o…
▽ More
Most periodicity search algorithms used in pulsar astronomy today are highly efficient and take advantage of multiple CPUs or GPUs. The bottlenecks are usually represented by the operations that require an informed choice from an expert eye. A typical case is the presence of radio-frequency interferences in the data, that often mimic the periodic signals of pulsars, and require visual inspection of hundreds or thousands of pulsar "candidates" satisfying a number of preselected criteria. CICLOPS is a citizen science project designed to transform the search for pulsars into an entertaining 3D video game. We build a distributed computing platform, running calculations with the user's CPUs and GPUs and using the unique human abilities in pattern recognition to find the best candidate pulsations.
△ Less
Submitted 23 December, 2020;
originally announced December 2020.
