Interactive Generation of Laparoscopic Videos with Diffusion Models
Authors:
Ivan Iliash,
Simeon Allmendinger,
Felix Meissen,
Niklas Kühl,
Daniel Rückert
Abstract:
Generative AI, in general, and synthetic visual data generation, in specific, hold much promise for benefiting surgical training by providing photorealism to simulation environments. Current training methods primarily rely on reading materials and observing live surgeries, which can be time-consuming and impractical. In this work, we take a significant step towards improving the training process.…
▽ More
Generative AI, in general, and synthetic visual data generation, in specific, hold much promise for benefiting surgical training by providing photorealism to simulation environments. Current training methods primarily rely on reading materials and observing live surgeries, which can be time-consuming and impractical. In this work, we take a significant step towards improving the training process. Specifically, we use diffusion models in combination with a zero-shot video diffusion method to interactively generate realistic laparoscopic images and videos by specifying a surgical action through text and guiding the generation with tool positions through segmentation masks. We demonstrate the performance of our approach using the publicly available Cholec dataset family and evaluate the fidelity and factual correctness of our generated images using a surgical action recognition model as well as the pixel-wise F1-score for the spatial control of tool generation. We achieve an FID of 38.097 and an F1-score of 0.71.
△ Less
Submitted 23 April, 2024;
originally announced June 2024.
Learnable real-time inference of molecular composition from diffuse spectroscopy of brain tissue
Authors:
Ivan Ezhov,
Kevin Scibilia,
Luca Giannoni,
Florian Kofler,
Ivan Iliash,
Felix Hsieh,
Suprosanna Shit,
Charly Caredda,
Fred Lange,
Ilias Tachtsidis,
Daniel Rueckert
Abstract:
Diffuse optical modalities such as broadband near-infrared spectroscopy (bNIRS) and hyperspectral imaging (HSI) represent a promising alternative for low-cost, non-invasive, and fast monitoring of functional and structural properties of living tissue. Particularly, the possibility of extracting the molecular composition of the tissue from the optical spectra in real-time deems the spectroscopy tec…
▽ More
Diffuse optical modalities such as broadband near-infrared spectroscopy (bNIRS) and hyperspectral imaging (HSI) represent a promising alternative for low-cost, non-invasive, and fast monitoring of functional and structural properties of living tissue. Particularly, the possibility of extracting the molecular composition of the tissue from the optical spectra in real-time deems the spectroscopy techniques as a unique diagnostic tool. However, no established method exists to streamline the inference of the biochemical composition from the optical spectrum for real-time applications such as surgical monitoring. In this paper, we analyse a machine learning technique for fast and accurate inference of changes in the molecular composition of brain tissue. We reconsider and propose modifications to the existing learnable methodology based on the Beer-Lambert law, which analytically connects the spectra with concentrations. We evaluate the method's applicability to linear and non-linear formulations of the Beer-Lambert law. The approach is tested on real data obtained from the bNIRS- and HSI-based optical monitoring of brain tissue. The results demonstrate that the proposed method enables real-time molecular composition inference while maintaining the accuracy of traditional linear and non-linear optimization solvers. Preliminary findings show that Beer-Lambert law-based spectral unmixing allows to contrast brain anatomy semantics such as the vessel tree and tumor area.
△ Less
Submitted 15 August, 2024; v1 submitted 27 September, 2023;
originally announced September 2023.
