-
Quark: Real-time, High-resolution, and General Neural View Synthesis
Authors:
John Flynn,
Michael Broxton,
Lukas Murmann,
Lucy Chai,
Matthew DuVall,
Clément Godard,
Kathryn Heal,
Srinivas Kaza,
Stephen Lombardi,
Xuan Luo,
Supreeth Achar,
Kira Prabhu,
Tiancheng Sun,
Lynn Tsai,
Ryan Overbeck
Abstract:
We present a novel neural algorithm for performing high-quality, high-resolution, real-time novel view synthesis. From a sparse set of input RGB images or videos streams, our network both reconstructs the 3D scene and renders novel views at 1080p resolution at 30fps on an NVIDIA A100. Our feed-forward network generalizes across a wide variety of datasets and scenes and produces state-of-the-art qu…
▽ More
We present a novel neural algorithm for performing high-quality, high-resolution, real-time novel view synthesis. From a sparse set of input RGB images or videos streams, our network both reconstructs the 3D scene and renders novel views at 1080p resolution at 30fps on an NVIDIA A100. Our feed-forward network generalizes across a wide variety of datasets and scenes and produces state-of-the-art quality for a real-time method. Our quality approaches, and in some cases surpasses, the quality of some of the top offline methods. In order to achieve these results we use a novel combination of several key concepts, and tie them together into a cohesive and effective algorithm. We build on previous works that represent the scene using semi-transparent layers and use an iterative learned render-and-refine approach to improve those layers. Instead of flat layers, our method reconstructs layered depth maps (LDMs) that efficiently represent scenes with complex depth and occlusions. The iterative update steps are embedded in a multi-scale, UNet-style architecture to perform as much compute as possible at reduced resolution. Within each update step, to better aggregate the information from multiple input views, we use a specialized Transformer-based network component. This allows the majority of the per-input image processing to be performed in the input image space, as opposed to layer space, further increasing efficiency. Finally, due to the real-time nature of our reconstruction and rendering, we dynamically create and discard the internal 3D geometry for each frame, generating the LDM for each view. Taken together, this produces a novel and effective algorithm for view synthesis. Through extensive evaluation, we demonstrate that we achieve state-of-the-art quality at real-time rates. Project page: https://quark-3d.github.io/
△ Less
Submitted 25 November, 2024;
originally announced November 2024.
-
DeepView: View Synthesis with Learned Gradient Descent
Authors:
John Flynn,
Michael Broxton,
Paul Debevec,
Matthew DuVall,
Graham Fyffe,
Ryan Overbeck,
Noah Snavely,
Richard Tucker
Abstract:
We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high…
▽ More
We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity. We show that our method achieves high-quality, state-of-the-art results on two datasets: the Kalantari light field dataset, and a new camera array dataset, Spaces, which we make publicly available.
△ Less
Submitted 17 June, 2019;
originally announced June 2019.
-
A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality
Authors:
Ryan S. Overbeck,
Daniel Erickson,
Daniel Evangelakos,
Matt Pharr,
Paul Debevec
Abstract:
We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets with two novel light field camera rigs designed for portable and efficient light field acquisition. We introduce a novel real-time light field reconstruction algorithm that uses a per-view geometry and a disk-based blend…
▽ More
We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets with two novel light field camera rigs designed for portable and efficient light field acquisition. We introduce a novel real-time light field reconstruction algorithm that uses a per-view geometry and a disk-based blending field. We also demonstrate how to use a light field prefiltering operation to project from a high-quality offline reconstruction model into our real-time model while suppressing artifacts. We introduce a practical approach for compressing light fields by modifying the VP9 video codec to provide high quality compression with real-time, random access decompression.
We combine these components into a complete light field system offering convenient acquisition, compact file size, and high-quality rendering while generating stereo views at 90Hz on commodity VR hardware. Using our system, we built a freely available light field experience application called Welcome to Light Fields featuring a library of panoramic light field stills for consumer VR which has been downloaded over 15,000 times.
△ Less
Submitted 20 October, 2018;
originally announced October 2018.
-
Multicritical behavior in dissipative Ising models
Authors:
Vincent R. Overbeck,
Mohammad F. Maghrebi,
Alexey V. Gorshkov,
Hendrik Weimer
Abstract:
We analyze theoretically the many-body dynamics of a dissipative Ising model in a transverse field using a variational approach. We find that the steady state phase diagram is substantially modified compared to its equilibrium counterpart, including the appearance of a multicritical point belonging to a different universality class. Building on our variational analysis, we establish a field-theore…
▽ More
We analyze theoretically the many-body dynamics of a dissipative Ising model in a transverse field using a variational approach. We find that the steady state phase diagram is substantially modified compared to its equilibrium counterpart, including the appearance of a multicritical point belonging to a different universality class. Building on our variational analysis, we establish a field-theoretical treatment corresponding to a dissipative variant of a Ginzburg-Landau theory, which allows us to compute the upper critical dimension of the system. Finally, we present a possible experimental realization of the dissipative Ising model using ultracold Rydberg gases.
△ Less
Submitted 28 September, 2017; v1 submitted 28 June, 2016;
originally announced June 2016.
-
Time evolution of open quantum many-body systems
Authors:
Vincent R. Overbeck,
Hendrik Weimer
Abstract:
We establish a generic method to analyze the time evolution of open quantum many-body systems. Our approach is based on a variational integration of the quantum master equation describing the dynamics and naturally connects to a variational principle for its nonequilibrium steady state. We successfully apply our variational method to study dissipative Rydberg gases, finding excellent quantitative…
▽ More
We establish a generic method to analyze the time evolution of open quantum many-body systems. Our approach is based on a variational integration of the quantum master equation describing the dynamics and naturally connects to a variational principle for its nonequilibrium steady state. We successfully apply our variational method to study dissipative Rydberg gases, finding excellent quantitative agreement with small-scale simulations of the full quantum master equation. We observe that correlations related to non-Markovian behavior play a significant role during the relaxation dynamics towards the steady state. We further quantify this non-Markovianity and find it to be closely connected to an information-theoretical measure of quantum and classical correlations.
△ Less
Submitted 28 September, 2017; v1 submitted 5 October, 2015;
originally announced October 2015.
