-
Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenes
Authors:
Dimitris Angelis,
Prodromos Kolyvakis,
Manos Kamarianakis,
George Papagiannakis
Abstract:
This paper introduces a novel integration of Large Language Models (LLMs) with Conformal Geometric Algebra (CGA) to revolutionize controllable 3D scene editing, particularly for object repositioning tasks, which traditionally requires intricate manual processes and specialized expertise. These conventional methods typically suffer from reliance on large training datasets or lack a formalized langu…
▽ More
This paper introduces a novel integration of Large Language Models (LLMs) with Conformal Geometric Algebra (CGA) to revolutionize controllable 3D scene editing, particularly for object repositioning tasks, which traditionally requires intricate manual processes and specialized expertise. These conventional methods typically suffer from reliance on large training datasets or lack a formalized language for precise edits. Utilizing CGA as a robust formal language, our system, shenlong, precisely models spatial transformations necessary for accurate object repositioning. Leveraging the zero-shot learning capabilities of pre-trained LLMs, shenlong translates natural language instructions into CGA operations which are then applied to the scene, facilitating exact spatial transformations within 3D scenes without the need for specialized pre-training. Implemented in a realistic simulation environment, shenlong ensures compatibility with existing graphics pipelines. To accurately assess the impact of CGA, we benchmark against robust Euclidean Space baselines, evaluating both latency and accuracy. Comparative performance evaluations indicate that shenlong significantly reduces LLM response times by 16% and boosts success rates by 9.6% on average compared to the traditional methods. Notably, shenlong achieves a 100% perfect success rate in common practical queries, a benchmark where other systems fall short. These advancements underscore shenlong's potential to democratize 3D scene editing, enhancing accessibility and fostering innovation across sectors such as education, digital entertainment, and virtual reality.
△ Less
Submitted 5 August, 2024;
originally announced August 2024.
-
Sample-efficient neural likelihood-free Bayesian inference of implicit HMMs
Authors:
Sanmitra Ghosh,
Paul J. Birrell,
Daniela De Angelis
Abstract:
Likelihood-free inference methods based on neural conditional density estimation were shown to drastically reduce the simulation burden in comparison to classical methods such as ABC. When applied in the context of any latent variable model, such as a Hidden Markov model (HMM), these methods are designed to only estimate the parameters, rather than the joint distribution of the parameters and the…
▽ More
Likelihood-free inference methods based on neural conditional density estimation were shown to drastically reduce the simulation burden in comparison to classical methods such as ABC. When applied in the context of any latent variable model, such as a Hidden Markov model (HMM), these methods are designed to only estimate the parameters, rather than the joint distribution of the parameters and the hidden states. Naive application of these methods to a HMM, ignoring the inference of this joint posterior distribution, will thus produce an inaccurate estimate of the posterior predictive distribution, in turn hampering the assessment of goodness-of-fit. To rectify this problem, we propose a novel, sample-efficient likelihood-free method for estimating the high-dimensional hidden states of an implicit HMM. Our approach relies on learning directly the intractable posterior distribution of the hidden states, using an autoregressive-flow, by exploiting the Markov property. Upon evaluating our approach on some implicit HMMs, we found that the quality of the estimates retrieved using our method is comparable to what can be achieved using a much more computationally expensive SMC algorithm.
△ Less
Submitted 2 May, 2024;
originally announced May 2024.
-
UniSG^GA: A 3D scenegraph powered by Geometric Algebra unifying geometry, behavior and GNNs towards generative AI
Authors:
Manos Kamarianakis,
Antonis Protopsaltis,
Dimitris Angelis,
Paul Zikas,
Mike Kentros,
George Papagiannakis
Abstract:
This work presents the introduction of UniSG^GA, a novel integrated scenegraph structure, that to incorporates behavior and geometry data on a 3D scene. It is specifically designed to seamlessly integrate Graph Neural Networks (GNNs) and address the challenges associated with transforming a 3D scenegraph (3D-SG) during generative tasks. To effectively capture and preserve the topological relations…
▽ More
This work presents the introduction of UniSG^GA, a novel integrated scenegraph structure, that to incorporates behavior and geometry data on a 3D scene. It is specifically designed to seamlessly integrate Graph Neural Networks (GNNs) and address the challenges associated with transforming a 3D scenegraph (3D-SG) during generative tasks. To effectively capture and preserve the topological relationships between objects in a simplified way, within the graph representation, we propose UniSG^GA, that seamlessly integrates Geometric Algebra (GA) forms. This novel approach enhances the overall performance and capability of GNNs in handling generative and predictive tasks, opening up new possibilities and aiming to lay the foundation for further exploration and development of graph-based generative AI models that can effectively incorporate behavior data for enhanced scene generation and synthesis.
△ Less
Submitted 18 June, 2023;
originally announced June 2023.
-
Project Elements: A computational entity-component-system in a scene-graph pythonic framework, for a neural, geometric computer graphics curriculum
Authors:
George Papagiannakis,
Manos Kamarianakis,
Antonis Protopsaltis,
Dimitris Angelis,
Paul Zikas
Abstract:
We present the Elements project, a lightweight, open-source, computational science and computer graphics (CG) framework, tailored for educational needs, that offers, for the first time, the advantages of an Entity-Component-System (ECS) along with the rapid prototyping convenience of a Scenegraph-based pythonic framework. This novelty allows advances in the teaching of CG: from heterogeneous direc…
▽ More
We present the Elements project, a lightweight, open-source, computational science and computer graphics (CG) framework, tailored for educational needs, that offers, for the first time, the advantages of an Entity-Component-System (ECS) along with the rapid prototyping convenience of a Scenegraph-based pythonic framework. This novelty allows advances in the teaching of CG: from heterogeneous directed acyclic graphs and depth-first traversals, to animation, skinning, geometric algebra and shader-based components rendered via unique systems all the way to their representation as graph neural networks for 3D scientific visualization. Taking advantage of the unique ECS in a a Scenegraph underlying system, this project aims to bridge CG curricula and modern game engines (MGEs), that are based on the same approach but often present these notions in a black-box approach. It is designed to actively utilize software design patterns, under an extensible open-source approach. Although Elements provides a modern (i.e., shader-based as opposed to fixed-function OpenGL), simple to program approach with Jupyter notebooks and unit-tests, its CG pipeline is not black-box, exposing for teaching for the first time unique challenging scientific, visual and neural computing concepts.
△ Less
Submitted 27 June, 2023; v1 submitted 15 February, 2023;
originally announced February 2023.
-
MAGES 4.0: Accelerating the world's transition to VR training and democratizing the authoring of the medical metaverse
Authors:
Paul Zikas,
Antonis Protopsaltis,
Nick Lydatakis,
Mike Kentros,
Stratos Geronikolakis,
Steve Kateros,
Manos Kamarianakis,
Giannis Evangelou,
Achilleas Filippidis,
Eleni Grigoriou,
Dimitris Angelis,
Michail Tamiolakis,
Michael Dodis,
George Kokiadis,
John Petropoulos,
Maria Pateraki,
George Papagiannakis
Abstract:
In this work, we propose MAGES 4.0, a novel Software Development Kit (SDK) to accelerate the creation of collaborative medical training applications in VR/AR. Our solution is essentially a low-code metaverse authoring platform for developers to rapidly prototype high-fidelity and high-complexity medical simulations. MAGES breaks the authoring boundaries across extended reality, since networked par…
▽ More
In this work, we propose MAGES 4.0, a novel Software Development Kit (SDK) to accelerate the creation of collaborative medical training applications in VR/AR. Our solution is essentially a low-code metaverse authoring platform for developers to rapidly prototype high-fidelity and high-complexity medical simulations. MAGES breaks the authoring boundaries across extended reality, since networked participants can also collaborate using different virtual/augmented reality as well as mobile and desktop devices, in the same metaverse world. With MAGES we propose an upgrade to the outdated 150-year-old master-apprentice medical training model. Our platform incorporates, in a nutsell, the following novelties: a) 5G edge-cloud remote rendering and physics dissection layer, b) realistic real-time simulation of organic tissues as soft-bodies under 10ms, c) a highly realistic cutting and tearing algorithm, d) neural network assessment for user profiling and, e) a VR recorder to record and replay or debrief the training simulation from any perspective.
△ Less
Submitted 19 January, 2023; v1 submitted 19 September, 2022;
originally announced September 2022.
-
Progressive tearing and cutting of soft-bodies in high-performance virtual reality
Authors:
Manos Kamarianakis,
Antonis Protopsaltis,
Dimitris Angelis,
Michail Tamiolakis,
George Papagiannakis
Abstract:
We present an algorithm that allows a user within a virtual environment to perform real-time unconstrained cuts or consecutive tears, i.e., progressive, continuous fractures on a deformable rigged and soft-body mesh model in high-performance 10ms. In order to recreate realistic results for different physically-principled materials such as sponges, hard or soft tissues, we incorporate a novel soft-…
▽ More
We present an algorithm that allows a user within a virtual environment to perform real-time unconstrained cuts or consecutive tears, i.e., progressive, continuous fractures on a deformable rigged and soft-body mesh model in high-performance 10ms. In order to recreate realistic results for different physically-principled materials such as sponges, hard or soft tissues, we incorporate a novel soft-body deformation, via a particle system layered on-top of a linear-blend skinning model. Our framework allows the simulation of realistic, surgical-grade cuts and continuous tears, especially valuable in the context of medical VR training. In order to achieve high performance in VR, our algorithms are based on Euclidean geometric predicates on the rigged mesh, without requiring any specific model pre-processing. The contribution of this work lies on the fact that current frameworks supporting similar kinds of model tearing, either do not operate in high-performance real-time or only apply to predefined tears. The framework presented allows the user to freely cut or tear a 3D mesh model in a consecutive way, under 10ms, while preserving its soft-body behaviour and/or allowing further animation.
△ Less
Submitted 18 September, 2022;
originally announced September 2022.
-
Assessing unconstrained surgical cuttings in VR using CNNs
Authors:
Ilias Chrysovergis,
Manos Kamarianakis,
Mike Kentros,
Dimitris Angelis,
Antonis Protopsaltis,
George Papagiannakis
Abstract:
We present a Convolutional Neural Network (CNN) suitable to assess unconstrained surgical cuttings, trained on a dataset created with a data augmentation technique.
We present a Convolutional Neural Network (CNN) suitable to assess unconstrained surgical cuttings, trained on a dataset created with a data augmentation technique.
△ Less
Submitted 2 May, 2022;
originally announced May 2022.
