Recommendations on test datasets for evaluating AI solutions in pathology
Authors:
André Homeyer,
Christian Geißler,
Lars Ole Schwen,
Falk Zakrzewski,
Theodore Evans,
Klaus Strohmenger,
Max Westphal,
Roman David Bülow,
Michaela Kargl,
Aray Karjauv,
Isidre Munné-Bertran,
Carl Orge Retzlaff,
Adrià Romero-López,
Tomasz Sołtysiński,
Markus Plass,
Rita Carvalho,
Peter Steinbach,
Yu-Chia Lan,
Nassim Bouteldja,
David Haber,
Mateo Rojas-Carulla,
Alireza Vafaei Sadr,
Matthias Kraft,
Daniel Krüger,
Rutger Fick
, et al. (5 additional authors not shown)
Abstract:
Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recom…
▽ More
Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing.
A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations for the collection of test datasets.
We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries?
The recommendations are intended to help AI developers demonstrate the utility of their products and to help regulatory agencies and end users verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.
△ Less
Submitted 21 April, 2022;
originally announced April 2022.
Quantification of Bore Path Uncertainty in Borehole Heat Exchanger Arrays
Authors:
Philipp Steinbach,
Daniel Otto Schulte,
Bastian Welsch,
Ingo Sass,
Jens Lang
Abstract:
Borehole heat exchanger arrays have become a common implement for the utilization of thermal energy in the soil. Building these facilities is expensive, especially the drilling of boreholes, into which closed-pipe heat exchangers are inserted. Therefore, cost-reducing drilling methods are common practice, which can produce inaccuracies of varying degree. This brings into question how much these in…
▽ More
Borehole heat exchanger arrays have become a common implement for the utilization of thermal energy in the soil. Building these facilities is expensive, especially the drilling of boreholes, into which closed-pipe heat exchangers are inserted. Therefore, cost-reducing drilling methods are common practice, which can produce inaccuracies of varying degree. This brings into question how much these inaccuracies could potentially affect the performance of a planned system. In the presented case study, an uncertainty quantification for seasonally operated borehole heat exchanger arrays is performed to analyze the bore paths' deviations impact. We introduce an adaptive, anisotropic stochastic collocation method, known as the generalized Smolyak algorithm, which was previously unused in this context and apply it to a numerical model of the borehole heat exchanger array. Our results show that the borehole heat exchanger array performance is surprisingly reliable even with potentially severe implementation errors during their construction. This, coupled with the potential uses of the presented method in similar applications gives planners and investors valuable information regarding the viability of borehole heat exchanger arrays in the face of uncertainty. With this paper, we hope to provide a powerful statistical tool to the field of geothermal energy, in which uncertainty quantification methods are still rarely used at this point. The discussed case study represents a jumping-off point for further investigations on the effects of uncertainty on borehole heat exchanger arrays and borehole thermal energy storage systems.
△ Less
Submitted 27 February, 2021;
originally announced March 2021.
