-
TetWeave: Isosurface Extraction using On-The-Fly Delaunay Tetrahedral Grids for Gradient-Based Mesh Optimization
Authors:
Alexandre Binninger,
Ruben Wiersma,
Philipp Herholz,
Olga Sorkine-Hornung
Abstract:
We introduce TetWeave, a novel isosurface representation for gradient-based mesh optimization that jointly optimizes the placement of a tetrahedral grid used for Marching Tetrahedra and a novel directional signed distance at each point. TetWeave constructs tetrahedral grids on-the-fly via Delaunay triangulation, enabling increased flexibility compared to predefined grids. The extracted meshes are…
▽ More
We introduce TetWeave, a novel isosurface representation for gradient-based mesh optimization that jointly optimizes the placement of a tetrahedral grid used for Marching Tetrahedra and a novel directional signed distance at each point. TetWeave constructs tetrahedral grids on-the-fly via Delaunay triangulation, enabling increased flexibility compared to predefined grids. The extracted meshes are guaranteed to be watertight, two-manifold and intersection-free. The flexibility of TetWeave enables a resampling strategy that places new points where reconstruction error is high and allows to encourage mesh fairness without compromising on reconstruction error. This leads to high-quality, adaptive meshes that require minimal memory usage and few parameters to optimize. Consequently, TetWeave exhibits near-linear memory scaling relative to the vertex count of the output mesh - a substantial improvement over predefined grids. We demonstrate the applicability of TetWeave to a broad range of challenging tasks in computer graphics and vision, such as multi-view 3D reconstruction, mesh compression and geometric texture generation.
△ Less
Submitted 8 May, 2025; v1 submitted 7 May, 2025;
originally announced May 2025.
-
Higher Order Continuity for Smooth As-Rigid-As-Possible Shape Modeling
Authors:
Annika Oehri,
Philipp Herholz,
Olga Sorkine-Hornung
Abstract:
We propose a modification of the As-Rigid-As-Possible (ARAP) mesh deformation energy with higher order smoothness, which overcomes a prominent limitation of the original ARAP formulation: spikes and lack of continuity at the manipulation handles. Our method avoids spikes even when using single-point positional constraints. Since no explicit rotations have to be specified, the user interaction can…
▽ More
We propose a modification of the As-Rigid-As-Possible (ARAP) mesh deformation energy with higher order smoothness, which overcomes a prominent limitation of the original ARAP formulation: spikes and lack of continuity at the manipulation handles. Our method avoids spikes even when using single-point positional constraints. Since no explicit rotations have to be specified, the user interaction can be realized through a simple click-and-drag interface, where points on the mesh can be selected and moved around while the rest of the mesh surface automatically deforms accordingly. Our method preserves the benefits of ARAP deformations: it is easy to implement and thus useful for practical applications, while its efficiency makes it usable in real-time, interactive scenarios on detailed models.
△ Less
Submitted 17 January, 2025;
originally announced January 2025.
-
Interactive Scene Authoring with Specialized Generative Primitives
Authors:
Clément Jambon,
Changwoon Choi,
Dongsu Zhang,
Olga Sorkine-Hornung,
Young Min Kim
Abstract:
Generating high-quality 3D digital assets often requires expert knowledge of complex design tools. We introduce Specialized Generative Primitives, a generative framework that allows non-expert users to author high-quality 3D scenes in a seamless, lightweight, and controllable manner. Each primitive is an efficient generative model that captures the distribution of a single exemplar from the real w…
▽ More
Generating high-quality 3D digital assets often requires expert knowledge of complex design tools. We introduce Specialized Generative Primitives, a generative framework that allows non-expert users to author high-quality 3D scenes in a seamless, lightweight, and controllable manner. Each primitive is an efficient generative model that captures the distribution of a single exemplar from the real world. With our framework, users capture a video of an environment, which we turn into a high-quality and explicit appearance model thanks to 3D Gaussian Splatting. Users then select regions of interest guided by semantically-aware features. To create a generative primitive, we adapt Generative Cellular Automata to single-exemplar training and controllable generation. We decouple the generative task from the appearance model by operating on sparse voxels and we recover a high-quality output with a subsequent sparse patch consistency step. Each primitive can be trained within 10 minutes and used to author new scenes interactively in a fully compositional manner. We showcase interactive sessions where various primitives are extracted from real-world scenes and controlled to create 3D assets and scenes in a few minutes. We also demonstrate additional capabilities of our primitives: handling various 3D representations to control generation, transferring appearances, and editing geometries.
△ Less
Submitted 19 December, 2024;
originally announced December 2024.
-
AIpparel: A Multimodal Foundation Model for Digital Garments
Authors:
Kiyohiro Nakayama,
Jan Ackermann,
Timur Levent Kesdogan,
Yang Zheng,
Maria Korosteleva,
Olga Sorkine-Hornung,
Leonidas J. Guibas,
Guandao Yang,
Gordon Wetzstein
Abstract:
Apparel is essential to human life, offering protection, mirroring cultural identities, and showcasing personal style. Yet, the creation of garments remains a time-consuming process, largely due to the manual work involved in designing them. To simplify this process, we introduce AIpparel, a multimodal foundation model for generating and editing sewing patterns. Our model fine-tunes state-of-the-a…
▽ More
Apparel is essential to human life, offering protection, mirroring cultural identities, and showcasing personal style. Yet, the creation of garments remains a time-consuming process, largely due to the manual work involved in designing them. To simplify this process, we introduce AIpparel, a multimodal foundation model for generating and editing sewing patterns. Our model fine-tunes state-of-the-art large multimodal models (LMMs) on a custom-curated large-scale dataset of over 120,000 unique garments, each with multimodal annotations including text, images, and sewing patterns. Additionally, we propose a novel tokenization scheme that concisely encodes these complex sewing patterns so that LLMs can learn to predict them efficiently. AIpparel achieves state-of-the-art performance in single-modal tasks, including text-to-garment and image-to-garment prediction, and enables novel multimodal garment generation applications such as interactive garment editing. The project website is at https://georgenakayama.github.io/AIpparel/.
△ Less
Submitted 5 April, 2025; v1 submitted 5 December, 2024;
originally announced December 2024.
-
SD-$π$XL: Generating Low-Resolution Quantized Imagery via Score Distillation
Authors:
Alexandre Binninger,
Olga Sorkine-Hornung
Abstract:
Low-resolution quantized imagery, such as pixel art, is seeing a revival in modern applications ranging from video game graphics to digital design and fabrication, where creativity is often bound by a limited palette of elemental units. Despite their growing popularity, the automated generation of quantized images from raw inputs remains a significant challenge, often necessitating intensive manua…
▽ More
Low-resolution quantized imagery, such as pixel art, is seeing a revival in modern applications ranging from video game graphics to digital design and fabrication, where creativity is often bound by a limited palette of elemental units. Despite their growing popularity, the automated generation of quantized images from raw inputs remains a significant challenge, often necessitating intensive manual input. We introduce SD-$π$XL, an approach for producing quantized images that employs score distillation sampling in conjunction with a differentiable image generator. Our method enables users to input a prompt and optionally an image for spatial conditioning, set any desired output size $H \times W$, and choose a palette of $n$ colors or elements. Each color corresponds to a distinct class for our generator, which operates on an $H \times W \times n$ tensor. We adopt a softmax approach, computing a convex sum of elements, thus rendering the process differentiable and amenable to backpropagation. We show that employing Gumbel-softmax reparameterization allows for crisp pixel art effects. Unique to our method is the ability to transform input images into low-resolution, quantized versions while retaining their key semantic features. Our experiments validate SD-$π$XL's performance in creating visually pleasing and faithful representations, consistently outperforming the current state-of-the-art. Furthermore, we showcase SD-$π$XL's practical utility in fabrication through its applications in interlocking brick mosaic, beading and embroidery design.
△ Less
Submitted 8 October, 2024;
originally announced October 2024.
-
WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds
Authors:
Peizhuo Li,
Sebastian Starke,
Yuting Ye,
Olga Sorkine-Hornung
Abstract:
We present a new approach for understanding the periodicity structure and semantics of motion datasets, independently of the morphology and skeletal structure of characters. Unlike existing methods using an overly sparse high-dimensional latent, we propose a phase manifold consisting of multiple closed curves, each corresponding to a latent amplitude. With our proposed vector quantized periodic au…
▽ More
We present a new approach for understanding the periodicity structure and semantics of motion datasets, independently of the morphology and skeletal structure of characters. Unlike existing methods using an overly sparse high-dimensional latent, we propose a phase manifold consisting of multiple closed curves, each corresponding to a latent amplitude. With our proposed vector quantized periodic autoencoder, we learn a shared phase manifold for multiple characters, such as a human and a dog, without any supervision. This is achieved by exploiting the discrete structure and a shallow network as bottlenecks, such that semantically similar motions are clustered into the same curve of the manifold, and the motions within the same component are aligned temporally by the phase variable. In combination with an improved motion matching framework, we demonstrate the manifold's capability of timing and semantics alignment in several applications, including motion retrieval, transfer and stylization. Code and pre-trained models for this paper are available at https://peizhuoli.github.io/walkthedog.
△ Less
Submitted 11 July, 2024;
originally announced July 2024.
-
Neural Garment Dynamics via Manifold-Aware Transformers
Authors:
Peizhuo Li,
Tuanfeng Y. Wang,
Timur Levent Kesdogan,
Duygu Ceylan,
Olga Sorkine-Hornung
Abstract:
Data driven and learning based solutions for modeling dynamic garments have significantly advanced, especially in the context of digital humans. However, existing approaches often focus on modeling garments with respect to a fixed parametric human body model and are limited to garment geometries that were seen during training. In this work, we take a different approach and model the dynamics of a…
▽ More
Data driven and learning based solutions for modeling dynamic garments have significantly advanced, especially in the context of digital humans. However, existing approaches often focus on modeling garments with respect to a fixed parametric human body model and are limited to garment geometries that were seen during training. In this work, we take a different approach and model the dynamics of a garment by exploiting its local interactions with the underlying human body. Specifically, as the body moves, we detect local garment-body collisions, which drive the deformation of the garment. At the core of our approach is a mesh-agnostic garment representation and a manifold-aware transformer network design, which together enable our method to generalize to unseen garment and body geometries. We evaluate our approach on a wide variety of garment types and motion sequences and provide competitive qualitative and quantitative results with respect to the state of the art.
△ Less
Submitted 13 May, 2024;
originally announced July 2024.
-
Dress Anyone : Automatic Physically-Based Garment Pattern Refitting
Authors:
Hsiao-yu Chen,
Egor Larionov,
Ladislav Kavan,
Gene Lin,
Doug Roble,
Olga Sorkine-Hornung,
Tuur Stuyck
Abstract:
Well-fitted clothing is essential for both real and virtual garments to enable self-expression and accurate representation for a large variety of body types. Common practice in the industry is to provide a pre-made selection of distinct garment sizes such as small, medium and large. While these may cater to certain groups of individuals that fall within this distribution, they often exclude large…
▽ More
Well-fitted clothing is essential for both real and virtual garments to enable self-expression and accurate representation for a large variety of body types. Common practice in the industry is to provide a pre-made selection of distinct garment sizes such as small, medium and large. While these may cater to certain groups of individuals that fall within this distribution, they often exclude large sections of the population. In contrast, individually tailored clothing offers a solution to obtain custom-fit garments that are tailored to each individual. However, manual tailoring is time-consuming and requires specialized knowledge, prohibiting the approach from being applied to produce fitted clothing at scale. To address this challenge, we propose a novel method leveraging differentiable simulation for refitting and draping 3D garments and their corresponding 2D pattern panels onto a new body shape, enabling a workflow where garments only need to be designed once, in a single size, and they can be automatically refitted to support numerous body size and shape variations. Our method enables downstream applications, where our optimized 3D drape can be directly ingested into game engines or other applications. Our 2D sewing patterns allow for accurate physics-based simulations and enables manufacturing clothing for the real world.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
GarmentCodeData: A Dataset of 3D Made-to-Measure Garments With Sewing Patterns
Authors:
Maria Korosteleva,
Timur Levent Kesdogan,
Fabian Kemper,
Stephan Wenninger,
Jasmin Koller,
Yuhan Zhang,
Mario Botsch,
Olga Sorkine-Hornung
Abstract:
Recent research interest in the learning-based processing of garments, from virtual fitting to generation and reconstruction, stumbles on a scarcity of high-quality public data in the domain. We contribute to resolving this need by presenting the first large-scale synthetic dataset of 3D made-to-measure garments with sewing patterns, as well as its generation pipeline. GarmentCodeData contains 115…
▽ More
Recent research interest in the learning-based processing of garments, from virtual fitting to generation and reconstruction, stumbles on a scarcity of high-quality public data in the domain. We contribute to resolving this need by presenting the first large-scale synthetic dataset of 3D made-to-measure garments with sewing patterns, as well as its generation pipeline. GarmentCodeData contains 115,000 data points that cover a variety of designs in many common garment categories: tops, shirts, dresses, jumpsuits, skirts, pants, etc., fitted to a variety of body shapes sampled from a custom statistical body model based on CAESAR, as well as a standard reference body shape, applying three different textile materials. To enable the creation of datasets of such complexity, we introduce a set of algorithms for automatically taking tailor's measures on sampled body shapes, sampling strategies for sewing pattern design, and propose an automatic, open-source 3D garment draping pipeline based on a fast XPBD simulator, while contributing several solutions for collision resolution and drape correctness to enable scalability.
Project Page: https://igl.ethz.ch/projects/GarmentCodeData/
△ Less
Submitted 5 September, 2024; v1 submitted 27 May, 2024;
originally announced May 2024.
-
Computational Smocking through Fabric-Thread Interaction
Authors:
Ningfeng Zhou,
Jing Ren,
Olga Sorkine-Hornung
Abstract:
We formalize Italian smocking, an intricate embroidery technique that gathers flat fabric into pleats along meandering lines of stitches, resulting in pleats that fold and gather where the stitching veers. In contrast to English smocking, characterized by colorful stitches decorating uniformly shaped pleats, and Canadian smocking, which uses localized knots to form voluminous pleats, Italian smock…
▽ More
We formalize Italian smocking, an intricate embroidery technique that gathers flat fabric into pleats along meandering lines of stitches, resulting in pleats that fold and gather where the stitching veers. In contrast to English smocking, characterized by colorful stitches decorating uniformly shaped pleats, and Canadian smocking, which uses localized knots to form voluminous pleats, Italian smocking permits the fabric to move freely along the stitched threads following curved paths, resulting in complex and unpredictable pleats with highly diverse, irregular structures, achieved simply by pulling on the threads. We introduce a novel method for digital previewing of Italian smocking results, given the thread stitching path as input. Our method uses a coarse-grained mass-spring system to simulate the interaction between the threads and the fabric. This configuration guides the fine-level fabric deformation through an adaptation of the state-of-the-art simulator, C-IPC. Our method models the general problem of fabric-thread interaction and can be readily adapted to preview Canadian smocking as well. We compare our results to baseline approaches and physical fabrications to demonstrate the accuracy of our method.
△ Less
Submitted 16 January, 2024; v1 submitted 10 January, 2024;
originally announced January 2024.
-
Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior
Authors:
Qingqing Zhao,
Peizhuo Li,
Wang Yifan,
Olga Sorkine-Hornung,
Gordon Wetzstein
Abstract:
Creating believable motions for various characters has long been a goal in computer graphics. Current learning-based motion synthesis methods depend on extensive motion datasets, which are often challenging, if not impossible, to obtain. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancement…
▽ More
Creating believable motions for various characters has long been a goal in computer graphics. Current learning-based motion synthesis methods depend on extensive motion datasets, which are often challenging, if not impossible, to obtain. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we utilize this alternative data source and introduce a neural motion synthesis approach through retargeting. Our method generates plausible motions for characters that have only pose data by transferring motion from an existing motion capture dataset of another character, which can have drastically different skeletons. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist-created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Project page: https://cyanzhao42.github.io/pose2motion
△ Less
Submitted 31 October, 2023;
originally announced October 2023.
-
Digital 3D Smocking Design
Authors:
Jing Ren,
Aviv Segall,
Olga Sorkine-Hornung
Abstract:
We develop an optimization-based method to model smocking, a surface embroidery technique that provides decorative geometric texturing while maintaining stretch properties of the fabric. During smocking, multiple pairs of points on the fabric are stitched together, creating non-manifold geometric features and visually pleasing textures. Designing smocking patterns is challenging, because the outco…
▽ More
We develop an optimization-based method to model smocking, a surface embroidery technique that provides decorative geometric texturing while maintaining stretch properties of the fabric. During smocking, multiple pairs of points on the fabric are stitched together, creating non-manifold geometric features and visually pleasing textures. Designing smocking patterns is challenging, because the outcome of stitching is unpredictable: the final texture is often revealed only when the whole smocking process is completed, necessitating painstaking physical fabrication and time consuming trial-and-error experimentation. This motivates us to seek a digital smocking design method. Straightforward attempts to compute smocked fabric geometry using surface deformation or cloth simulation methods fail to produce realistic results, likely due to the intricate structure of the designs, the large number of contacts and high-curvature folds. We instead formulate smocking as a graph embedding and shape deformation problem. We extract a coarse graph representing the fabric and the stitching constraints, and then derive the graph structure of the smocked result. We solve for the 3D embedding of this graph, which in turn reliably guides the deformation of the high-resolution fabric mesh. Our optimization based method is simple, efficient, and flexible, which allows us to build an interactive system for smocking pattern exploration. To demonstrate the accuracy of our method, we compare our results to real fabrications on a large set of smocking patterns
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling
Authors:
Alexandre Binninger,
Amir Hertz,
Olga Sorkine-Hornung,
Daniel Cohen-Or,
Raja Giryes
Abstract:
We present SENS, a novel method for generating and editing 3D models from hand-drawn sketches, including those of abstract nature. Our method allows users to quickly and easily sketch a shape, and then maps the sketch into the latent space of a part-aware neural implicit shape architecture. SENS analyzes the sketch and encodes its parts into ViT patch encoding, subsequently feeding them into a tra…
▽ More
We present SENS, a novel method for generating and editing 3D models from hand-drawn sketches, including those of abstract nature. Our method allows users to quickly and easily sketch a shape, and then maps the sketch into the latent space of a part-aware neural implicit shape architecture. SENS analyzes the sketch and encodes its parts into ViT patch encoding, subsequently feeding them into a transformer decoder that converts them to shape embeddings suitable for editing 3D neural implicit shapes. SENS provides intuitive sketch-based generation and editing, and also succeeds in capturing the intent of the user's sketch to generate a variety of novel and expressive 3D shapes, even from abstract and imprecise sketches. Additionally, SENS supports refinement via part reconstruction, allowing for nuanced adjustments and artifact removal. It also offers part-based modeling capabilities, enabling the combination of features from multiple sketches to create more complex and customized 3D shapes. We demonstrate the effectiveness of our model compared to the state-of-the-art using objective metric evaluation criteria and a user study, both indicating strong performance on sketches with a medium level of abstraction. Furthermore, we showcase our method's intuitive sketch-based shape editing capabilities, and validate it through a usability study.
△ Less
Submitted 21 February, 2024; v1 submitted 9 June, 2023;
originally announced June 2023.
-
GarmentCode: Programming Parametric Sewing Patterns
Authors:
Maria Korosteleva,
Olga Sorkine-Hornung
Abstract:
Garment modeling is an essential task of the global apparel industry and a core part of digital human modeling. Realistic representation of garments with valid sewing patterns is key to their accurate digital simulation and eventual fabrication. However, little-to-no computational tools provide support for bridging the gap between high-level construction goals and low-level editing of pattern geom…
▽ More
Garment modeling is an essential task of the global apparel industry and a core part of digital human modeling. Realistic representation of garments with valid sewing patterns is key to their accurate digital simulation and eventual fabrication. However, little-to-no computational tools provide support for bridging the gap between high-level construction goals and low-level editing of pattern geometry, e.g., combining or switching garment elements, semantic editing, or design exploration that maintains the validity of a sewing pattern. We suggest the first DSL for garment modeling -- GarmentCode -- that applies principles of object-oriented programming to garment construction and allows designing sewing patterns in a hierarchical, component-oriented manner. The programming-based paradigm naturally provides unique advantages of component abstraction, algorithmic manipulation, and free-form design parametrization. We additionally support the construction process by automating typical low-level tasks like placing a dart at a desired location. In our prototype garment configurator, users can manipulate meaningful design parameters and body measurements, while the construction of pattern geometry is handled by garment programs implemented with GarmentCode. Our configurator enables the free exploration of rich design spaces and the creation of garments using interchangeable, parameterized components. We showcase our approach by producing a variety of garment designs and retargeting them to different body shapes using our configurator.
Project page: https://igl.ethz.ch/projects/garmentcode/
△ Less
Submitted 19 October, 2023; v1 submitted 6 June, 2023;
originally announced June 2023.
-
Example-based Motion Synthesis via Generative Motion Matching
Authors:
Weiyu Li,
Xuelin Chen,
Peizhuo Li,
Olga Sorkine-Hornung,
Baoquan Chen
Abstract:
We present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion M…
▽ More
We present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly. Code and data for this paper are at https://wyysf-98.github.io/GenMM/
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
UVDoc: Neural Grid-based Document Unwarping
Authors:
Floor Verhoeven,
Tanguy Magne,
Olga Sorkine-Hornung
Abstract:
Restoring the original, flat appearance of a printed document from casual photographs of bent and wrinkled pages is a common everyday problem. In this paper we propose a novel method for grid-based single-image document unwarping. Our method performs geometric distortion correction via a fully convolutional deep neural network that learns to predict the 3D grid mesh of the document and the corresp…
▽ More
Restoring the original, flat appearance of a printed document from casual photographs of bent and wrinkled pages is a common everyday problem. In this paper we propose a novel method for grid-based single-image document unwarping. Our method performs geometric distortion correction via a fully convolutional deep neural network that learns to predict the 3D grid mesh of the document and the corresponding 2D unwarping grid in a dual-task fashion, implicitly encoding the coupling between the shape of a 3D piece of paper and its 2D image. In order to allow unwarping models to train on data that is more realistic in appearance than the commonly used synthetic Doc3D dataset, we create and publish our own dataset, called UVDoc, which combines pseudo-photorealistic document images with physically accurate 3D shape and unwarping function annotations. Our dataset is labeled with all the information necessary to train our unwarping network, without having to engineer separate loss functions that can deal with the lack of ground-truth typically found in document in the wild datasets. We perform an in-depth evaluation that demonstrates that with the inclusion of our novel pseudo-photorealistic dataset, our relatively small network architecture achieves state-of-the-art results on the DocUNet benchmark. We show that the pseudo-photorealistic nature of our UVDoc dataset allows for new and better evaluation methods, such as lighting-corrected MS-SSIM. We provide a novel benchmark dataset that facilitates such evaluations, and propose a metric that quantifies line straightness after unwarping. Our code, results and UVDoc dataset are available at https://github.com/tanguymagne/UVDoc.
△ Less
Submitted 27 February, 2024; v1 submitted 6 February, 2023;
originally announced February 2023.
-
Smooth Non-Rigid Shape Matching via Effective Dirichlet Energy Optimization
Authors:
Robin Magnet,
Jing Ren,
Olga Sorkine-Hornung,
Maks Ovsjanikov
Abstract:
We introduce pointwise map smoothness via the Dirichlet energy into the functional map pipeline, and propose an algorithm for optimizing it efficiently, which leads to high-quality results in challenging settings. Specifically, we first formulate the Dirichlet energy of the pulled-back shape coordinates, as a way to evaluate smoothness of a pointwise map across discrete surfaces. We then extend th…
▽ More
We introduce pointwise map smoothness via the Dirichlet energy into the functional map pipeline, and propose an algorithm for optimizing it efficiently, which leads to high-quality results in challenging settings. Specifically, we first formulate the Dirichlet energy of the pulled-back shape coordinates, as a way to evaluate smoothness of a pointwise map across discrete surfaces. We then extend the recently proposed discrete solver and show how a strategy based on auxiliary variable reformulation allows us to optimize pointwise map smoothness alongside desirable functional map properties such as bijectivity. This leads to an efficient map refinement strategy that simultaneously improves functional and point-to-point correspondences, obtaining smooth maps even on non-isometric shape pairs. Moreover, we demonstrate that several previously proposed methods for computing smooth maps can be reformulated as variants of our approach, which allows us to compare different formulations in a consistent framework. Finally, we compare these methods both on existing benchmarks and on a new rich dataset that we introduce, which contains non-rigid, non-isometric shape pairs with inter-category and cross-category correspondences. Our work leads to a general framework for optimizing and analyzing map smoothness both conceptually and in challenging practical settings.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
MoDi: Unconditional Motion Synthesis from Diverse Data
Authors:
Sigal Raab,
Inbal Leibovitch,
Peizhuo Li,
Kfir Aberman,
Olga Sorkine-Hornung,
Daniel Cohen-Or
Abstract:
The emergence of neural networks has revolutionized the field of motion synthesis. Yet, learning to unconditionally synthesize motions from a given distribution remains challenging, especially when the motions are highly diverse. In this work, we present MoDi -- a generative model trained in an unsupervised setting from an extremely diverse, unstructured and unlabeled dataset. During inference, Mo…
▽ More
The emergence of neural networks has revolutionized the field of motion synthesis. Yet, learning to unconditionally synthesize motions from a given distribution remains challenging, especially when the motions are highly diverse. In this work, we present MoDi -- a generative model trained in an unsupervised setting from an extremely diverse, unstructured and unlabeled dataset. During inference, MoDi can synthesize high-quality, diverse motions. Despite the lack of any structure in the dataset, our model yields a well-behaved and highly structured latent space, which can be semantically clustered, constituting a strong motion prior that facilitates various applications including semantic editing and crowd simulation. In addition, we present an encoder that inverts real motions into MoDi's natural motion manifold, issuing solutions to various ill-posed challenges such as completion from prefix and spatial editing. Our qualitative and quantitative experiments achieve state-of-the-art results that outperform recent SOTA techniques. Code and trained models are available at https://sigal-raab.github.io/MoDi.
△ Less
Submitted 18 December, 2022; v1 submitted 16 June, 2022;
originally announced June 2022.
-
GANimator: Neural Motion Synthesis from a Single Sequence
Authors:
Peizhuo Li,
Kfir Aberman,
Zihan Zhang,
Rana Hanocka,
Olga Sorkine-Hornung
Abstract:
We present GANimator, a generative model that learns to synthesize novel motions from a single, short motion sequence. GANimator generates motions that resemble the core elements of the original motion, while simultaneously synthesizing novel and diverse movements. Existing data-driven techniques for motion synthesis require a large motion dataset which contains the desired and specific skeletal s…
▽ More
We present GANimator, a generative model that learns to synthesize novel motions from a single, short motion sequence. GANimator generates motions that resemble the core elements of the original motion, while simultaneously synthesizing novel and diverse movements. Existing data-driven techniques for motion synthesis require a large motion dataset which contains the desired and specific skeletal structure. By contrast, GANimator only requires training on a single motion sequence, enabling novel motion synthesis for a variety of skeletal structures e.g., bipeds, quadropeds, hexapeds, and more. Our framework contains a series of generative and adversarial neural networks, each responsible for generating motions in a specific frame rate. The framework progressively learns to synthesize motion from random noise, enabling hierarchical control over the generated motion content across varying levels of detail. We show a number of applications, including crowd simulation, key-frame editing, style transfer, and interactive control, which all learn from a single input sequence. Code and data for this paper are at https://peizhuoli.github.io/ganimator.
△ Less
Submitted 5 May, 2022;
originally announced May 2022.
-
Computational Pattern Making from 3D Garment Models
Authors:
Nico Pietroni,
Corentin Dumery,
Raphael Guenot-Falque,
Mark Liu,
Teresa Vidal-Calleja,
Olga Sorkine-Hornung
Abstract:
We propose a method for computing a sewing pattern of a given 3D garment model. Our algorithm segments an input 3D garment shape into patches and computes their 2D parameterization, resulting in pattern pieces that can be cut out of fabric and sewn together to manufacture the garment. Unlike the general state-of-the-art approaches for surface cutting and flattening, our method explicitly targets g…
▽ More
We propose a method for computing a sewing pattern of a given 3D garment model. Our algorithm segments an input 3D garment shape into patches and computes their 2D parameterization, resulting in pattern pieces that can be cut out of fabric and sewn together to manufacture the garment. Unlike the general state-of-the-art approaches for surface cutting and flattening, our method explicitly targets garment fabrication. It accounts for the unique properties and constraints of tailoring, such as seam symmetry, the usage of darts, fabric grain alignment, and a flattening distortion measure that models woven fabric deformation, respecting its anisotropic behavior. We bootstrap a recent patch layout approach developed for quadrilateral remeshing and adapt it to the purpose of computational pattern making, ensuring that the deformation of each pattern piece stays within prescribed bounds of cloth stress. While our algorithm can automatically produce the sewing patterns, it is fast enough to admit user input to creatively iterate on the pattern design. Our method can take several target poses of the 3D garment into account and integrate them into the sewing pattern design. We demonstrate results on both skintight and loose garments, showcasing the versatile application possibilities of our approach.
△ Less
Submitted 21 February, 2022;
originally announced February 2022.
-
SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation
Authors:
Amir Hertz,
Or Perel,
Raja Giryes,
Olga Sorkine-Hornung,
Daniel Cohen-Or
Abstract:
Neural implicit fields are quickly emerging as an attractive representation for learning based techniques. However, adopting them for 3D shape modeling and editing is challenging. We introduce a method for $\mathbf{E}$diting $\mathbf{I}$mplicit $\mathbf{S}$hapes $\mathbf{T}$hrough $\mathbf{P}$art $\mathbf{A}$ware $\mathbf{G}$enera$\mathbf{T}$ion, permuted in short as SPAGHETTI. Our architecture al…
▽ More
Neural implicit fields are quickly emerging as an attractive representation for learning based techniques. However, adopting them for 3D shape modeling and editing is challenging. We introduce a method for $\mathbf{E}$diting $\mathbf{I}$mplicit $\mathbf{S}$hapes $\mathbf{T}$hrough $\mathbf{P}$art $\mathbf{A}$ware $\mathbf{G}$enera$\mathbf{T}$ion, permuted in short as SPAGHETTI. Our architecture allows for manipulation of implicit shapes by means of transforming, interpolating and combining shape segments together, without requiring explicit part supervision. SPAGHETTI disentangles shape part representation into extrinsic and intrinsic geometric information. This characteristic enables a generative framework with part-level control. The modeling capabilities of SPAGHETTI are demonstrated using an interactive graphical interface, where users can directly edit neural implicit shapes.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
Sparsity-Specific Code Optimization using Expression Trees
Authors:
Philipp Herholz,
Xuan Tang,
Teseo Schneider,
Shoaib Kamil,
Daniele Panozzo,
Olga Sorkine-Hornung
Abstract:
We introduce a code generator that converts unoptimized C++ code operating on sparse data into vectorized and parallel CPU or GPU kernels. Our approach unrolls the computation into a massive expression graph, performs redundant expression elimination, grouping, and then generates an architecture-specific kernel to solve the same problem, assuming that the sparsity pattern is fixed, which is a comm…
▽ More
We introduce a code generator that converts unoptimized C++ code operating on sparse data into vectorized and parallel CPU or GPU kernels. Our approach unrolls the computation into a massive expression graph, performs redundant expression elimination, grouping, and then generates an architecture-specific kernel to solve the same problem, assuming that the sparsity pattern is fixed, which is a common scenario in many applications in computer graphics and scientific computing. We show that our approach scales to large problems and can achieve speedups of two orders of magnitude on CPUs and three orders of magnitude on GPUs, compared to a set of manually optimized CPU baselines. To demonstrate the practical applicability of our approach, we employ it to optimize popular algorithms with applications to physical simulation and interactive mesh deformation.
△ Less
Submitted 14 March, 2022; v1 submitted 15 October, 2021;
originally announced October 2021.
-
Mesh Draping: Parametrization-Free Neural Mesh Transfer
Authors:
Amir Hertz,
Or Perel,
Raja Giryes,
Olga Sorkine-Hornung,
Daniel Cohen-Or
Abstract:
Despite recent advances in geometric modeling, 3D mesh modeling still involves a considerable amount of manual labor by experts. In this paper, we introduce Mesh Draping: a neural method for transferring existing mesh structure from one shape to another. The method drapes the source mesh over the target geometry and at the same time seeks to preserve the carefully designed characteristics of the s…
▽ More
Despite recent advances in geometric modeling, 3D mesh modeling still involves a considerable amount of manual labor by experts. In this paper, we introduce Mesh Draping: a neural method for transferring existing mesh structure from one shape to another. The method drapes the source mesh over the target geometry and at the same time seeks to preserve the carefully designed characteristics of the source mesh. At its core, our method deforms the source mesh using progressive positional encoding. We show that by leveraging gradually increasing frequencies to guide the neural optimization, we are able to achieve stable and high quality mesh transfer. Our approach is simple and requires little user guidance, compared to contemporary surface mapping techniques which rely on parametrization or careful manual tuning. Most importantly, Mesh Draping is a parameterization-free method, and thus applicable to a variety of target shape representations, including point clouds, polygon soups, and non-manifold meshes. We demonstrate that the transferred meshing remains faithful to the source mesh design characteristics, and at the same time fits the target geometry well.
△ Less
Submitted 11 October, 2021;
originally announced October 2021.
-
Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields
Authors:
Wang Yifan,
Lukas Rahmann,
Olga Sorkine-Hornung
Abstract:
We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high frequency signal is constrained geometrically b…
▽ More
We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high frequency signal is constrained geometrically by the low frequency signal. Importantly, this disentanglement is unsupervised thanks to a tailored architectural design that has an innate frequency hierarchy by construction. We explore implicit displacement field surface reconstruction and detail transfer and demonstrate superior representational power, training stability and generalizability.
△ Less
Submitted 2 February, 2022; v1 submitted 9 June, 2021;
originally announced June 2021.
-
Learning Skeletal Articulations with Neural Blend Shapes
Authors:
Peizhuo Li,
Kfir Aberman,
Rana Hanocka,
Libin Liu,
Olga Sorkine-Hornung,
Baoquan Chen
Abstract:
Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters…
▽ More
Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure which produces high quality pose dependent deformations. Our framework learns to rig and skin characters with the same articulation structure (e.g., bipeds or quadrupeds), and builds the desired skeleton hierarchy into the network architecture. Furthermore, we propose neural blend shapes--a set of corrective pose-dependent shapes which improve the deformation quality in the joint regions in order to address the notorious artifacts resulting from standard rigging and skinning. Our system estimates neural blend shapes for input meshes with arbitrary connectivity, as well as weighting coefficients which are conditioned on the input joint rotations. Unlike recent deep learning techniques which supervise the network with ground-truth rigging and skinning parameters, our approach does not assume that the training data has a specific underlying deformation model. Instead, during training, the network observes deformed shapes and learns to infer the corresponding rig, skin and blend shapes using indirect supervision. During inference, we demonstrate that our network generalizes to unseen characters with arbitrary mesh connectivity, including unrigged characters built by 3D artists. Conforming to standard skeletal animation models enables direct plug-and-play in standard animation software, as well as game engines.
△ Less
Submitted 6 May, 2021;
originally announced May 2021.
-
SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization
Authors:
Amir Hertz,
Or Perel,
Raja Giryes,
Olga Sorkine-Hornung,
Daniel Cohen-Or
Abstract:
Multilayer-perceptrons (MLP) are known to struggle with learning functions of high-frequencies, and in particular cases with wide frequency bands. We present a spatially adaptive progressive encoding (SAPE) scheme for input signals of MLP networks, which enables them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. SAP…
▽ More
Multilayer-perceptrons (MLP) are known to struggle with learning functions of high-frequencies, and in particular cases with wide frequency bands. We present a spatially adaptive progressive encoding (SAPE) scheme for input signals of MLP networks, which enables them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. SAPE gradually unmasks signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of SAPE on a variety of domains and applications, including regression of low dimensional signals and images, representation learning of occupancy networks, and a geometric task of mesh transfer between 3D shapes.
△ Less
Submitted 28 May, 2021; v1 submitted 19 April, 2021;
originally announced April 2021.
-
Dev2PQ: Planar Quadrilateral Strip Remeshing of Developable Surfaces
Authors:
Floor Verhoeven,
Amir Vaxman,
Tim Hoffmann,
Olga Sorkine-Hornung
Abstract:
We introduce an algorithm to remesh triangle meshes representing developable surfaces to planar quad dominant meshes. The output of our algorithm consists of planar quadrilateral (PQ) strips that are aligned to principal curvature directions and closely approximate the curved parts of the input developable, and planar polygons representing the flat parts of the input. Developable PQ-strip meshes a…
▽ More
We introduce an algorithm to remesh triangle meshes representing developable surfaces to planar quad dominant meshes. The output of our algorithm consists of planar quadrilateral (PQ) strips that are aligned to principal curvature directions and closely approximate the curved parts of the input developable, and planar polygons representing the flat parts of the input. Developable PQ-strip meshes are useful in many areas of shape modeling, thanks to the simplicity of fabrication from flat sheet material. Unfortunately, they are difficult to model due to their restrictive combinatorics and locking issues. Other representations of developable surfaces, such as arbitrary triangle or quad meshes, are more suitable for interactive freeform modeling, but generally have non-planar faces or are not aligned to principal curvatures. Our method leverages the modeling flexibility of non-ruling based representations of developable surfaces, while still obtaining developable, curvature aligned PQ-strip meshes. Our algorithm optimizes for a scalar function on the input mesh, such that its level sets are extrinsically straight and align well to the locally estimated ruling directions. The condition that guarantees straight level sets is nonlinear of high order and numerically difficult to enforce in a straightforward manner. We devise an alternating optimization method that makes our problem tractable and practical to compute. Our method works automatically on any developable input, including multiple patches and curved folds, without explicit domain decomposition. We demonstrate the effectiveness of our approach on a variety of developable surfaces and show how our remeshing can be used alongside handle based interactive freeform modeling of developable shapes.
△ Less
Submitted 24 June, 2021; v1 submitted 27 February, 2021;
originally announced March 2021.
-
Designing Personalized Garments with Body Movement
Authors:
Katja Wolff,
Philipp Herholz,
Verena Ziegler,
Frauke Link,
Nico Brügel,
Olga Sorkine-Hornung
Abstract:
The standardized sizes used in the garment industry do not cover the range of individual differences in body shape for most people, leading to ill-fitting clothes, high return rates and overproduction. Recent research efforts in both industry and academia therefore focus on virtual try-on and on-demand fabrication of individually fitting garments. We propose an interactive design tool for creating…
▽ More
The standardized sizes used in the garment industry do not cover the range of individual differences in body shape for most people, leading to ill-fitting clothes, high return rates and overproduction. Recent research efforts in both industry and academia therefore focus on virtual try-on and on-demand fabrication of individually fitting garments. We propose an interactive design tool for creating custom-fit garments based on 3D body scans of the intended wearer. Our method explicitly incorporates transitions between various body poses to ensure a better fit and freedom of movement. The core of our method focuses on tools to create a 3D garment shape directly on an avatar without an underlying sewing pattern, and on the adjustment of that garment's rest shape while interpolating and moving through the different input poses. We alternate between cloth simulation and rest shape adjustment based on stretch to achieve the final shape of the garment. At any step in the real-time process, we allow for interactive changes to the garment. Once the garment shape is finalized for production, established techniques can be used to parameterize it into a 2D sewing pattern or transform it into a knitting pattern.
△ Less
Submitted 19 September, 2022; v1 submitted 10 February, 2021;
originally announced February 2021.
-
Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations
Authors:
Wang Yifan,
Shihao Wu,
Cengiz Oztireli,
Olga Sorkine-Hornung
Abstract:
Neural implicit functions have emerged as a powerful representation for surfaces in 3D. Such a function can encode a high quality surface with intricate details into the parameters of a deep neural network. However, optimizing for the parameters for accurate and robust reconstructions remains a challenge, especially when the input data is noisy or incomplete. In this work, we develop a hybrid neur…
▽ More
Neural implicit functions have emerged as a powerful representation for surfaces in 3D. Such a function can encode a high quality surface with intricate details into the parameters of a deep neural network. However, optimizing for the parameters for accurate and robust reconstructions remains a challenge, especially when the input data is noisy or incomplete. In this work, we develop a hybrid neural surface representation that allows us to impose geometry-aware sampling and regularization, which significantly improves the fidelity of reconstructions. We propose to use \emph{iso-points} as an explicit representation for a neural implicit function. These points are computed and updated on-the-fly during training to capture important geometric features and impose geometric constraints on the optimization. We demonstrate that our method can be adopted to improve state-of-the-art techniques for reconstructing neural implicit surfaces from multi-view images or point clouds. Quantitative and qualitative evaluations show that, compared with existing sampling and optimization methods, our approach allows faster convergence, better generalization, and accurate recovery of details and topology.
△ Less
Submitted 9 April, 2021; v1 submitted 11 December, 2020;
originally announced December 2020.
-
Skeleton-Aware Networks for Deep Motion Retargeting
Authors:
Kfir Aberman,
Peizhuo Li,
Dani Lischinski,
Olga Sorkine-Hornung,
Daniel Cohen-Or,
Baoquan Chen
Abstract:
We introduce a novel deep learning framework for data-driven motion retargeting between skeletons, which may have different structure, yet corresponding to homeomorphic graphs. Importantly, our approach learns how to retarget without requiring any explicit pairing between the motions in the training set. We leverage the fact that different homeomorphic skeletons may be reduced to a common primal s…
▽ More
We introduce a novel deep learning framework for data-driven motion retargeting between skeletons, which may have different structure, yet corresponding to homeomorphic graphs. Importantly, our approach learns how to retarget without requiring any explicit pairing between the motions in the training set. We leverage the fact that different homeomorphic skeletons may be reduced to a common primal skeleton by a sequence of edge merging operations, which we refer to as skeletal pooling. Thus, our main technical contribution is the introduction of novel differentiable convolution, pooling, and unpooling operators. These operators are skeleton-aware, meaning that they explicitly account for the skeleton's hierarchical structure and joint adjacency, and together they serve to transform the original motion into a collection of deep temporal features associated with the joints of the primal skeleton. In other words, our operators form the building blocks of a new deep motion processing framework that embeds the motion into a common latent space, shared by a collection of homeomorphic skeletons. Thus, retargeting can be achieved simply by encoding to, and decoding from this latent space. Our experiments show the effectiveness of our framework for motion retargeting, as well as motion processing in general, compared to existing approaches. Our approach is also quantitatively evaluated on a synthetic dataset that contains pairs of motions applied to different skeletons. To the best of our knowledge, our method is the first to perform retargeting between skeletons with differently sampled kinematic chains, without any paired examples.
△ Less
Submitted 12 May, 2020;
originally announced May 2020.
-
Neural Cages for Detail-Preserving 3D Deformations
Authors:
Wang Yifan,
Noam Aigerman,
Vladimir G. Kim,
Siddhartha Chaudhuri,
Olga Sorkine-Hornung
Abstract:
We propose a novel learnable representation for detail-preserving shape deformation. The goal of our method is to warp a source shape to match the general structure of a target shape, while preserving the surface details of the source. Our method extends a traditional cage-based deformation technique, where the source shape is enclosed by a coarse control mesh termed \emph{cage}, and translations…
▽ More
We propose a novel learnable representation for detail-preserving shape deformation. The goal of our method is to warp a source shape to match the general structure of a target shape, while preserving the surface details of the source. Our method extends a traditional cage-based deformation technique, where the source shape is enclosed by a coarse control mesh termed \emph{cage}, and translations prescribed on the cage vertices are interpolated to any point on the source mesh via special weight functions. The use of this sparse cage scaffolding enables preserving surface details regardless of the shape's intricacy and topology. Our key contribution is a novel neural network architecture for predicting deformations by controlling the cage. We incorporate a differentiable cage-based deformation module in our architecture, and train our network end-to-end. Our method can be trained with common collections of 3D models in an unsupervised fashion, without any cage-specific annotations. We demonstrate the utility of our method for synthesizing shape variations and deformation transfer.
△ Less
Submitted 18 March, 2020; v1 submitted 13 December, 2019;
originally announced December 2019.
-
Differentiable Surface Splatting for Point-based Geometry Processing
Authors:
Wang Yifan,
Felice Serena,
Shihao Wu,
Cengiz Öztireli,
Olga Sorkine-Hornung
Abstract:
We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geome…
▽ More
We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geometry synthesis and denoising, where large scale topological changes, as well as small scale detail modifications, are accurately and robustly handled without requiring explicit connectivity, outperforming state-of-the-art techniques. The data and code are at https://github.com/yifita/DSS.
△ Less
Submitted 3 September, 2019; v1 submitted 10 June, 2019;
originally announced June 2019.
-
Patch-based Progressive 3D Point Set Upsampling
Authors:
Wang Yifan,
Shihao Wu,
Hui Huang,
Daniel Cohen-Or,
Olga Sorkine-Hornung
Abstract:
We present a detail-driven deep neural network for point set upsampling. A high-resolution point set is essential for point-based rendering and surface reconstruction. Inspired by the recent success of neural image super-resolution techniques, we progressively train a cascade of patch-based upsampling networks on different levels of detail end-to-end. We propose a series of architectural design co…
▽ More
We present a detail-driven deep neural network for point set upsampling. A high-resolution point set is essential for point-based rendering and surface reconstruction. Inspired by the recent success of neural image super-resolution techniques, we progressively train a cascade of patch-based upsampling networks on different levels of detail end-to-end. We propose a series of architectural design contributions that lead to a substantial performance boost. The effect of each technical contribution is demonstrated in an ablation study. Qualitative and quantitative experiments show that our method significantly outperforms the state-of-the-art learning-based and optimazation-based approaches, both in terms of handling low-resolution inputs and revealing high-fidelity details.
△ Less
Submitted 21 March, 2019; v1 submitted 27 November, 2018;
originally announced November 2018.
-
Deformation Capture via Soft and Stretchable Sensor Arrays
Authors:
Oliver Glauser,
Daniele Panozzo,
Otmar Hilliges,
Olga Sorkine-Hornung
Abstract:
We propose a hardware and software pipeline to fabricate flexible wearable sensors and use them to capture deformations without line of sight. Our first contribution is a low-cost fabrication pipeline to embed multiple aligned conductive layers with complex geometries into silicone compounds. Overlapping conductive areas from separate layers form local capacitors that measure dense area changes. C…
▽ More
We propose a hardware and software pipeline to fabricate flexible wearable sensors and use them to capture deformations without line of sight. Our first contribution is a low-cost fabrication pipeline to embed multiple aligned conductive layers with complex geometries into silicone compounds. Overlapping conductive areas from separate layers form local capacitors that measure dense area changes. Contrary to existing fabrication methods, the proposed technique only requires hardware that is readily available in modern fablabs. While area measurements alone are not enough to reconstruct the full 3D deformation of a surface, they become sufficient when paired with a data-driven prior. A novel semi-automatic tracking algorithm, based on an elastic surface geometry deformation, allows to capture ground-truth data with an optical mocap system, even under heavy occlusions or partially unobservable markers. The resulting dataset is used to train a regressor based on deep neural networks, directly mapping the area readings to global positions of surface vertices. We demonstrate the flexibility and accuracy of the proposed hardware and software in a series of controlled experiments, and design a prototype of wearable wrist, elbow and biceps sensors, which do not require line-of-sight and can be worn below regular clothing.
△ Less
Submitted 22 March, 2019; v1 submitted 11 April, 2018;
originally announced April 2018.
-
A Fully Progressive Approach to Single-Image Super-Resolution
Authors:
Yifan Wang,
Federico Perazzi,
Brian McWilliams,
Alexander Sorkine-Hornung,
Olga Sorkine-Hornung,
Christopher Schroers
Abstract:
Recent deep learning approaches to single image super-resolution have achieved impressive results in terms of traditional error measures and perceptual quality. However, in each case it remains challenging to achieve high quality results for large upsampling factors. To this end, we propose a method (ProSR) that is progressive both in architecture and training: the network upsamples an image in in…
▽ More
Recent deep learning approaches to single image super-resolution have achieved impressive results in terms of traditional error measures and perceptual quality. However, in each case it remains challenging to achieve high quality results for large upsampling factors. To this end, we propose a method (ProSR) that is progressive both in architecture and training: the network upsamples an image in intermediate steps, while the learning process is organized from easy to hard, as is done in curriculum learning. To obtain more photorealistic results, we design a generative adversarial network (GAN), named ProGanSR, that follows the same progressive multi-scale design principle. This not only allows to scale well to high upsampling factors (e.g., 8x) but constitutes a principled multi-scale approach that increases the reconstruction quality for all upsampling factors simultaneously. In particular ProSR ranks 2nd in terms of SSIM and 4th in terms of PSNR in the NTIRE2018 SISR challenge [34]. Compared to the top-ranking team, our model is marginally lower, but runs 5 times faster.
△ Less
Submitted 10 April, 2018; v1 submitted 9 April, 2018;
originally announced April 2018.
-
Shape Representation by Zippable Ribbons
Authors:
Christian Schüller,
Roi Poranne,
Olga Sorkine-Hornung
Abstract:
Shape fabrication from developable parts is the basis for arts such as papercraft and needlework, as well as modern architecture and CAD in general, and it has inspired much research. We observe that the assembly of complex 3D shapes created by existing methods often requires first fabricating many small flat parts and then carefully following instructions to assemble them together. Despite its si…
▽ More
Shape fabrication from developable parts is the basis for arts such as papercraft and needlework, as well as modern architecture and CAD in general, and it has inspired much research. We observe that the assembly of complex 3D shapes created by existing methods often requires first fabricating many small flat parts and then carefully following instructions to assemble them together. Despite its significance, this error prone and tedious process is generally neglected in the discussion. We propose an approach for shape representation through a single developable part that attaches to itself and requires no assembly instructions. Our inspiration comes from the so-called zipit bags, which are made of a single, long ribbon with a zipper around its boundary. In order to "assemble" the bag, one simply needs to zip up the ribbon. Our method operates in the same fashion, but it can be used to approximate any shape. Given a 3D model, our algorithm produces plans for a single 2D shape that can be laser cut in few parts from flat fabric or paper. We can then attach a zipper along the boundary for quick assembly and disassembly, or apply more traditional approaches, such as gluing and stitching. We show physical and virtual results that demonstrate the capabilities of our method and the ease with which shapes can be assembled.
△ Less
Submitted 7 November, 2017;
originally announced November 2017.
-
Discrete Geodesic Nets for Modeling Developable Surfaces
Authors:
Michael Rabinovich,
Tim Hoffmann,
Olga Sorkine-Hornung
Abstract:
We present a discrete theory for modeling developable surfaces as quadrilateral meshes satisfying simple angle constraints. The basis of our model is a lesser known characterization of developable surfaces as manifolds that can be parameterized through orthogonal geodesics. Our model is simple, local, and, unlike previous works, it does not directly encode the surface rulings. This allows us to mo…
▽ More
We present a discrete theory for modeling developable surfaces as quadrilateral meshes satisfying simple angle constraints. The basis of our model is a lesser known characterization of developable surfaces as manifolds that can be parameterized through orthogonal geodesics. Our model is simple, local, and, unlike previous works, it does not directly encode the surface rulings. This allows us to model continuous deformations of discrete developable surfaces independently of their decomposition into torsal and planar patches or the surface topology. We prove and experimentally demonstrate strong ties to smooth developable surfaces, including a theorem stating that every sampling of the smooth counterpart satisfies our constraints up to second order. We further present an extension of our model that enables a local definition of discrete isometry. We demonstrate the effectiveness of our discrete model in a developable surface editing system, as well as computation of an isometric interpolation between isometric discrete developable shapes.
△ Less
Submitted 26 July, 2017;
originally announced July 2017.
-
Consistently Orienting Facets in Polygon Meshes by Minimizing the Dirichlet Energy of Generalized Winding Numbers
Authors:
Kenshi Takayama,
Alec Jacobson,
Ladislav Kavan,
Olga Sorkine-Hornung
Abstract:
Jacobson et al. [JKSH13] hypothesized that the local coherency of the generalized winding number function could be used to correctly determine consistent facet orientations in polygon meshes. We report on an approach to consistently orienting facets in polygon meshes by minimizing the Dirichlet energy of generalized winding numbers. While the energy can be concisely formulated and efficiently comp…
▽ More
Jacobson et al. [JKSH13] hypothesized that the local coherency of the generalized winding number function could be used to correctly determine consistent facet orientations in polygon meshes. We report on an approach to consistently orienting facets in polygon meshes by minimizing the Dirichlet energy of generalized winding numbers. While the energy can be concisely formulated and efficiently computed, we found that this approach is fundamentally flawed and is unfortunately not applicable for most handmade meshes shared on popular mesh repositories such as Google 3D Warehouse.
△ Less
Submitted 23 June, 2014; v1 submitted 20 June, 2014;
originally announced June 2014.
