-
Artificial Blur Effect for Optical See-through Near-Eye Displays
Authors:
Shiva Sinaei,
Daisuke Iwai,
Kousuke Sato
Abstract:
Saliency modulation has significant potential for various applications. In our pursuit of implementing saliency modulation for optical see-through near-eye displays, we decided to introduce a blur effect to reduce the sharpness of specific areas while preserving the sharpness of others. In this study, we used a digital micromirror device (DMD) to separate the incoming light from a scene into sharp…
▽ More
Saliency modulation has significant potential for various applications. In our pursuit of implementing saliency modulation for optical see-through near-eye displays, we decided to introduce a blur effect to reduce the sharpness of specific areas while preserving the sharpness of others. In this study, we used a digital micromirror device (DMD) to separate the incoming light from a scene into sharp and blurred areas. To achieve this, we integrated an electrically tunable lens (ETL), which operates in its zero optical power mode when the reflected light from the DMD represents the sharp area (i.e., the blur area is masked). Conversely, when the reflected light indicates the blur area, the ETL adjusts to non-zero optical powers. Importantly, these modulations occur at a speed that surpasses the critical flicker frequency threshold of the human eye. Furthermore, we proposed an algorithm to mitigate the artifacts around the border area between the sharp and blur areas that are caused by the magnification of the ETL. We have also developed a prototype system to demonstrate the feasibility of our method.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands
Authors:
Yotam Erel,
Or Kozlovsky-Mordenfeld,
Daisuke Iwai,
Kosuke Sato,
Amit H. Bermano
Abstract:
We present a technique for dynamically projecting 3D content onto human hands with short perceived motion-to-photon latency. Computing the pose and shape of human hands accurately and quickly is a challenging task due to their articulated and deformable nature. We combine a slower 3D coarse estimation of the hand pose with high speed 2D correction steps which improve the alignment of the projectio…
▽ More
We present a technique for dynamically projecting 3D content onto human hands with short perceived motion-to-photon latency. Computing the pose and shape of human hands accurately and quickly is a challenging task due to their articulated and deformable nature. We combine a slower 3D coarse estimation of the hand pose with high speed 2D correction steps which improve the alignment of the projection to the hands, increase the projected surface area, and reduce perceived latency. Since our approach leverages a full 3D reconstruction of the hands, any arbitrary texture or reasonably performant effect can be applied, which was not possible before. We conducted two user studies to assess the benefits of using our method. The results show subjects are less sensitive to latency artifacts and perform faster and with more ease a given associated task over the naive approach of directly projecting rendered frames from the 3D pose estimation. We demonstrate several novel use cases and applications.
△ Less
Submitted 6 September, 2024;
originally announced September 2024.
-
Projection Mapping under Environmental Lighting by Replacing Room Lights with Heterogeneous Projectors
Authors:
Masaki Takeuchi,
Hiroki Kusuyama,
Daisuke Iwai,
Kosuke Sato
Abstract:
Projection mapping (PM) is a technique that enhances the appearance of real-world surfaces using projected images, enabling multiple people to view augmentations simultaneously, thereby facilitating communication and collaboration. However, PM typically requires a dark environment to achieve high-quality projections, limiting its practicality. In this paper, we overcome this limitation by replacin…
▽ More
Projection mapping (PM) is a technique that enhances the appearance of real-world surfaces using projected images, enabling multiple people to view augmentations simultaneously, thereby facilitating communication and collaboration. However, PM typically requires a dark environment to achieve high-quality projections, limiting its practicality. In this paper, we overcome this limitation by replacing conventional room lighting with heterogeneous projectors. These projectors replicate environmental lighting by selectively illuminating the scene, excluding the projection target. Our contributions include a distributed projector optimization framework designed to effectively replicate environmental lighting and the incorporation of a large-aperture projector, in addition to standard projectors, to reduce high-luminance emitted rays and hard shadows -- undesirable factors for collaborative tasks in PM. We conducted a series of quantitative and qualitative experiments, including user studies, to validate our approach. Our findings demonstrate that our projector-based lighting system significantly enhances the contrast and realism of PM results even under environmental lighting compared to typical lights. Furthermore, our method facilitates a substantial shift in the perceived color mode from the undesirable aperture-color mode, where observers perceive the projected object as self-luminous, to the surface-color mode in PM.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
Topology Optimization with Text-Guided Stylization
Authors:
Shengze Zhong,
Parinya Punpongsanon,
Daisuke Iwai,
Kosuke Sato
Abstract:
We propose an approach for the generation of topology-optimized structures with text-guided appearance stylization. This methodology aims to enrich the concurrent design of a structure's physical functionality and aesthetic appearance. Users can effortlessly input descriptive text to govern the style of the structure. Our system employs a hash-encoded neural network as the implicit structure repre…
▽ More
We propose an approach for the generation of topology-optimized structures with text-guided appearance stylization. This methodology aims to enrich the concurrent design of a structure's physical functionality and aesthetic appearance. Users can effortlessly input descriptive text to govern the style of the structure. Our system employs a hash-encoded neural network as the implicit structure representation backbone, which serves as the foundation for the co-optimization of structural mechanical performance, style, and connectivity, to ensure full-color, high-quality 3D-printable solutions. We substantiate the effectiveness of our system through extensive comparisons, demonstrations, and a 3D printing test.
△ Less
Submitted 24 October, 2023;
originally announced October 2023.
-
Focal Surface Projection: Extending Projector Depth-of-Field Using a Phase-Only Spatial Light Modulator
Authors:
Fumitaka Ueda,
Yuta Kageyama,
Daisuke Iwai,
Kosuke Sato
Abstract:
We present a focal surface projection to solve the narrow depth-of-field problem in projection mapping applications. We apply a phase-only spatial light modulator to realize nonuniform focusing distances, whereby the projected contents appear focused on a surface with considerable depth variations. The feasibility of the proposed technique was validated through a physical experiment.
We present a focal surface projection to solve the narrow depth-of-field problem in projection mapping applications. We apply a phase-only spatial light modulator to realize nonuniform focusing distances, whereby the projected contents appear focused on a surface with considerable depth variations. The feasibility of the proposed technique was validated through a physical experiment.
△ Less
Submitted 5 September, 2023;
originally announced September 2023.
-
Neural Projection Mapping Using Reflectance Fields
Authors:
Yotam Erel,
Daisuke Iwai,
Amit H. Bermano
Abstract:
We introduce a high resolution spatially adaptive light source, or a projector, into a neural reflectance field that allows to both calibrate the projector and photo realistic light editing. The projected texture is fully differentiable with respect to all scene parameters, and can be optimized to yield a desired appearance suitable for applications in augmented reality and projection mapping. Our…
▽ More
We introduce a high resolution spatially adaptive light source, or a projector, into a neural reflectance field that allows to both calibrate the projector and photo realistic light editing. The projected texture is fully differentiable with respect to all scene parameters, and can be optimized to yield a desired appearance suitable for applications in augmented reality and projection mapping. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. Using an analytical BRDF model and carefully selected projection patterns, our acquisition process is simple and intuitive, featuring a fixed uncalibrated projected and a handheld camera with a co-located light source. As we demonstrate, the virtual projector incorporated into the pipeline improves scene understanding and enables various projection mapping applications, alleviating the need for time consuming calibration steps performed in a traditional setting per view or projector location. In addition to enabling novel viewpoint synthesis, we demonstrate state-of-the-art performance projector compensation for novel viewpoints, improvement over the baselines in material and scene reconstruction, and three simply implemented scenarios where projection image optimization is performed, including the use of a 2D generative model to consistently dictate scene appearance from multiple viewpoints. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.
△ Less
Submitted 11 June, 2023;
originally announced June 2023.
-
NSTO: Neural Synthesizing Topology Optimization for Modulated Structure Generation
Authors:
Shengze Zhong,
Parinya Punpongsanon,
Daisuke Iwai,
Kosuke Sato
Abstract:
Nature evolves structures like honeycombs at optimized performance with limited material. These efficient structures can be artificially created with the collaboration of structural topology optimization and additive manufacturing. However, the extensive computation cost of topology optimization causes low mesh resolution, long solving time, and rough boundaries that fail to match the requirements…
▽ More
Nature evolves structures like honeycombs at optimized performance with limited material. These efficient structures can be artificially created with the collaboration of structural topology optimization and additive manufacturing. However, the extensive computation cost of topology optimization causes low mesh resolution, long solving time, and rough boundaries that fail to match the requirements for meeting the growing personal fabrication demands and printing capability. Therefore, we propose the neural synthesizing topology optimization that leverages a self-supervised coordinate-based network to optimize structures with significantly shorter computation time, where the network encodes the structural material layout as an implicit function of coordinates. Continuous solution space is further generated from optimization tasks under varying boundary conditions or constraints for users' instant inference of novel solutions. We demonstrate the system's efficacy for a broad usage scenario through numerical experiments and 3D printing.
△ Less
Submitted 21 March, 2023;
originally announced March 2023.
-
Is Projection Mapping Natural? Towards Physical World Augmentation Consistent with Light Field Context
Authors:
Daisuke Iwai
Abstract:
Projection mapping seamlessly merges real and virtual worlds. Although much effort was made to improve its image qualities so far, projection mapping is still unnatural. We introduce the first steps towards natural projection mapping by making the projection results consistent with the light field context of our daily scene.
Projection mapping seamlessly merges real and virtual worlds. Although much effort was made to improve its image qualities so far, projection mapping is still unnatural. We introduce the first steps towards natural projection mapping by making the projection results consistent with the light field context of our daily scene.
△ Less
Submitted 1 December, 2021;
originally announced December 2021.
-
Multifocal Stereoscopic Projection Mapping
Authors:
Sorashi Kimura,
Daisuke Iwai,
Parinya Punpongsanon,
Kosuke Sato
Abstract:
Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence--accommodation conflict (VAC). Therefore, we…
▽ More
Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence--accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the "virtual image'" (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: (1) The 3D CG object is displayed on non-planar and even moving surfaces; (2) the physical surfaces need to be shown without the focus modulation; (3) the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the "lens breathing" artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task.
△ Less
Submitted 8 October, 2021;
originally announced October 2021.
-
Directionally Decomposing Structured Light for Projector Calibration
Authors:
Masatoki Sugimoto,
Daisuke Iwai,
Koki Ishida,
Parinya Punpongsanon,
Kosuke Sato
Abstract:
Intrinsic projector calibration is essential in projection mapping (PM) applications, especially in dynamic PM. However, due to the shallow depth-of-field (DOF) of a projector, more work is needed to ensure accurate calibration. We aim to estimate the intrinsic parameters of a projector while avoiding the limitation of shallow DOF. As the core of our technique, we present a practical calibration d…
▽ More
Intrinsic projector calibration is essential in projection mapping (PM) applications, especially in dynamic PM. However, due to the shallow depth-of-field (DOF) of a projector, more work is needed to ensure accurate calibration. We aim to estimate the intrinsic parameters of a projector while avoiding the limitation of shallow DOF. As the core of our technique, we present a practical calibration device that requires a minimal working volume directly in front of the projector lens regardless of the projector's focusing distance and aperture size. The device consists of a flat-bed scanner and pinhole-array masks. For calibration, a projector projects a series of structured light patterns in the device. The pinholes directionally decompose the structured light, and only the projected rays that pass through the pinholes hit the scanner plane. For each pinhole, we extract a ray passing through the optical center of the projector. Consequently, we regard the projector as a pinhole projector that projects the extracted rays only, and we calibrate the projector by applying the standard camera calibration technique, which assumes a pinhole camera model. Using a proof-of-concept prototype, we demonstrate that our technique can calibrate projectors with different focusing distances and aperture sizes at the same accuracy as a conventional method. Finally, we confirm that our technique can provide intrinsic parameters accurate enough for a dynamic PM application, even when a projector is placed too far from a projection target for a conventional method to calibrate the projector using a fiducial object of reasonable size.
△ Less
Submitted 8 October, 2021;
originally announced October 2021.
-
Projector Pixel Redirection Using Phase-Only Spatial Light Modulator
Authors:
Haruka Terai,
Daisuke Iwai,
Kosuke Sato
Abstract:
In projection mapping from a projector to a non-planar surface, the pixel density on the surface becomes uneven. This causes the critical problem of local spatial resolution degradation. We confirmed that the pixel density uniformity on the surface was improved by redirecting projected rays using a phase-only spatial light modulator.
In projection mapping from a projector to a non-planar surface, the pixel density on the surface becomes uneven. This causes the critical problem of local spatial resolution degradation. We confirmed that the pixel density uniformity on the surface was improved by redirecting projected rays using a phase-only spatial light modulator.
△ Less
Submitted 11 July, 2021;
originally announced July 2021.
-
A Projector-Camera System Using Hybrid Pixels with Projection and Capturing Capabilities
Authors:
Kenta Yamamoto,
Daisuke Iwai,
Kosuke Sato
Abstract:
We propose a novel projector-camera system (ProCams) in which each pixel has both projection and capturing capabilities. Our proposed ProCams solves the difficulty of obtaining precise pixel correspondence between the projector and the camera. We implemented a proof-of-concept ProCams prototype and demonstrated its applicability to a dynamic projection mapping.
We propose a novel projector-camera system (ProCams) in which each pixel has both projection and capturing capabilities. Our proposed ProCams solves the difficulty of obtaining precise pixel correspondence between the projector and the camera. We implemented a proof-of-concept ProCams prototype and demonstrated its applicability to a dynamic projection mapping.
△ Less
Submitted 11 July, 2021;
originally announced July 2021.
-
IlluminatedFocus: Vision Augmentation using Spatial Defocusing via Focal Sweep Eyeglasses and High-Speed Projector
Authors:
Tatsuyuki Ueda,
Daisuke Iwai,
Takefumi Hiraki,
Kosuke Sato
Abstract:
Aiming at realizing novel vision augmentation experiences, this paper proposes the IlluminatedFocus technique, which spatially defocuses real-world appearances regardless of the distance from the user's eyes to observed real objects. With the proposed technique, a part of a real object in an image appears blurred, while the fine details of the other part at the same distance remain visible. We app…
▽ More
Aiming at realizing novel vision augmentation experiences, this paper proposes the IlluminatedFocus technique, which spatially defocuses real-world appearances regardless of the distance from the user's eyes to observed real objects. With the proposed technique, a part of a real object in an image appears blurred, while the fine details of the other part at the same distance remain visible. We apply Electrically Focus-Tunable Lenses (ETL) as eyeglasses and a synchronized high-speed projector as illumination for a real scene. We periodically modulate the focal lengths of the glasses (focal sweep) at more than 60 Hz so that a wearer cannot perceive the modulation. A part of the scene to appear focused is illuminated by the projector when it is in focus of the user's eyes, while another part to appear blurred is illuminated when it is out of the focus. As the basis of our spatial focus control, we build mathematical models to predict the range of distance from the ETL within which real objects become blurred on the retina of a user. Based on the blur range, we discuss a design guideline for effective illumination timing and focal sweep range. We also model the apparent size of a real scene altered by the focal length modulation. This leads to an undesirable visible seam between focused and blurred areas. We solve this unique problem by gradually blending the two areas. Finally, we demonstrate the feasibility of our proposal by implementing various vision augmentation applications.
△ Less
Submitted 6 February, 2020;
originally announced February 2020.
-
FibAR: Embedding Optical Fibers in 3D Printed Objects for Active Markers in Dynamic Projection Mapping
Authors:
Daiki Tone,
Daisuke Iwai,
Shinsaku Hiura,
Kosuke Sato
Abstract:
This paper presents a novel active marker for dynamic projection mapping (PM) that emits a temporal blinking pattern of infrared (IR) light representing its ID. We used a multi-material three dimensional (3D) printer to fabricate a projection object with optical fibers that can guide IR light from LEDs attached on the bottom of the object. The aperture of an optical fiber is typically very small;…
▽ More
This paper presents a novel active marker for dynamic projection mapping (PM) that emits a temporal blinking pattern of infrared (IR) light representing its ID. We used a multi-material three dimensional (3D) printer to fabricate a projection object with optical fibers that can guide IR light from LEDs attached on the bottom of the object. The aperture of an optical fiber is typically very small; thus, it is unnoticeable to human observers under projection and can be placed on a strongly curved part of a projection surface. In addition, the working range of our system can be larger than previous marker-based methods as the blinking patterns can theoretically be recognized by a camera placed at a wide range of distances from markers. We propose an automatic marker placement algorithm to spread multiple active markers over the surface of a projection object such that its pose can be robustly estimated using captured images from arbitrary directions. We also propose an optimization framework for determining the routes of the optical fibers in such a way that collisions of the fibers can be avoided while minimizing the loss of light intensity in the fibers. Through experiments conducted using three fabricated objects containing strongly curved surfaces, we confirmed that the proposed method can achieve accurate dynamic PMs in a significantly wide working range.
△ Less
Submitted 6 February, 2020;
originally announced February 2020.
-
Projection Mapping Technologies for AR
Authors:
Daisuke Iwai
Abstract:
This invited talk will present recent projection mapping technologies for augmented reality. First, fundamental technologies are briefly explained, which have been proposed to overcome the technical limitations of ordinary projectors. Second, augmented reality (AR) applications using projection mapping technologies are introduced.
This invited talk will present recent projection mapping technologies for augmented reality. First, fundamental technologies are briefly explained, which have been proposed to overcome the technical limitations of ordinary projectors. Second, augmented reality (AR) applications using projection mapping technologies are introduced.
△ Less
Submitted 10 April, 2017;
originally announced April 2017.
-
Procams-Based Cybernetics
Authors:
Kosuke Sato,
Daisuke Iwai,
Sei Ikeda,
Noriko Takemura
Abstract:
Procams-based cybernetics is a unique, emerging research field, which aims at enhancing and supporting our activities by naturally connecting human and computers/machines as a cooperative integrated system via projector-camera systems (procams). It rests on various research domains such as virtual/augmented reality, computer vision, computer graphics, projection display, human computer interface,…
▽ More
Procams-based cybernetics is a unique, emerging research field, which aims at enhancing and supporting our activities by naturally connecting human and computers/machines as a cooperative integrated system via projector-camera systems (procams). It rests on various research domains such as virtual/augmented reality, computer vision, computer graphics, projection display, human computer interface, human robot interaction and so on. This laboratory presentation provides a brief history including recent achievements of our procams-based cybernetics project.
△ Less
Submitted 9 October, 2015;
originally announced October 2015.
