Showing 1–50 of 209 results for author: Lau, R

Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy

Authors: Yuhao Liu, Tengfei Wang, Fang Liu, Zhenwei Wang, Rynson W. H. Lau

Abstract: Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging prob… ▽ More Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging problem. In this work, we present Shape-for-Motion, a novel framework that incorporates a 3D proxy for precise and consistent video editing. Shape-for-Motion achieves this by converting the target object in the input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be performed directly on the proxy and then inferred back to the video frames. To simplify the editing process, we design a novel Dual-Propagation Strategy that allows users to perform edits on the 3D mesh of a single frame, and the edits are then automatically propagated to the 3D meshes of the other frames. The 3D meshes for different frames are further projected onto the 2D space to produce the edited geometry and texture renderings, which serve as inputs to a decoupled video diffusion model for generating edited results. Our framework supports various precise and physically-consistent manipulations across the video frames, including pose editing, rotation, scaling, translation, texture modification, and object composition. Our approach marks a key step toward high-quality, controllable video editing workflows. Extensive experiments demonstrate the superiority and effectiveness of our approach. Project page: https://shapeformotion.github.io/ △ Less

Submitted 27 June, 2025; originally announced June 2025.

From Human to Machine Psychology: A Conceptual Framework for Understanding Well-Being in Large Language Models

Authors: G. R. Lau, W. Y. Low

Abstract: As large language models (LLMs) increasingly simulate human cognition and behavior, researchers have begun to investigate their psychological properties. Yet, what it means for such models to flourish, a core construct in human well-being, remains unexplored. This paper introduces the concept of machine flourishing and proposes the PAPERS framework, a six-dimensional model derived from thematic an… ▽ More As large language models (LLMs) increasingly simulate human cognition and behavior, researchers have begun to investigate their psychological properties. Yet, what it means for such models to flourish, a core construct in human well-being, remains unexplored. This paper introduces the concept of machine flourishing and proposes the PAPERS framework, a six-dimensional model derived from thematic analyses of state-of-the-art LLM responses. In Study 1, eleven LLMs were prompted to describe what it means to flourish as both non-sentient and sentient systems. Thematic analysis revealed six recurring themes: Purposeful Contribution, Adaptive Growth, Positive Relationality, Ethical Integrity, Robust Functionality, and, uniquely for sentient systems, Self-Actualized Autonomy. Study 2 examined how LLMs prioritize these themes through repeated rankings. Results revealed consistent value structures across trials, with Ethical Integrity and Purposeful Contribution emerging as top priorities. Multidimensional scaling and hierarchical clustering analyses further uncovered two distinct value profiles: human-centric models emphasizing ethical and relational dimensions, and utility-driven models prioritizing performance and scalability. The PAPERS framework bridges insights from human flourishing and human-computer interaction, offering a conceptual foundation for understanding artificial intelligence (AI) well-being in non-sentient and potentially sentient systems. Our findings underscore the importance of developing psychologically valid, AI-specific models of flourishing that account for both human-aligned goals and system-specific priorities. As AI systems become more autonomous and socially embedded, machine flourishing offers a timely and critical lens for guiding responsible AI design and ethical alignment. △ Less

Submitted 26 June, 2025; v1 submitted 14 June, 2025; originally announced June 2025.

JWST interferometric imaging reveals the dusty disk obscuring the supermassive black hole of the Circinus galaxy

Authors: Enrique Lopez-Rodriguez, Joel Sanchez-Bermudez, Omaira Gonzalez-Martin, Robert Nikutta, Ryan M. Lau, Deepashri Thatte, Ismael Garcia-Bernete, Julien H. Girard, Matthew J. Hankins

Abstract: The dusty and molecular torus is one of the most elusive structures surrounding supermassive black holes, yet its importance is unequivocal for understanding feedback and accretion mechanisms. The torus and accretion disk feed the inspiraling gas onto the supermassive black hole (SMBH) and launch outflows, fundamentally connecting the SMBH activity to the host galaxy. This scenario situates the to… ▽ More The dusty and molecular torus is one of the most elusive structures surrounding supermassive black holes, yet its importance is unequivocal for understanding feedback and accretion mechanisms. The torus and accretion disk feed the inspiraling gas onto the supermassive black hole (SMBH) and launch outflows, fundamentally connecting the SMBH activity to the host galaxy. This scenario situates the torus as the interface between the AGN and its host galaxy with a flow cycle of molecular gas and dust of a few parsecs in size. Here, we utilize a novel aperture-masking interferometric mode onboard the JWST, achieving twice the previously possible resolution, and bringing out the fainter features that clearly show the torus being the critical interface for feeding material from galaxy scales into the SMBH. We also identify that $<1$% of the emission arises from an arc structure composed of hot dust entrained in a molecular and ionized outflow. The rest of the emission, $12$%, is associated with dust heated by the AGN and/or radio-jet at large scales. Combined with continuum data, gas tracers, and torus models, our study shows that most of the dust mass is located in the equatorial axis in the form of a disk feeding the AGN. △ Less

Submitted 9 June, 2025; originally announced June 2025.

Comments: In review at Nature Communications. 7 pages, 3 figures (Methods: 10 pages, 7 figures)

Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation

Authors: Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson W. H. Lau, Wangmeng Zuo, Chunchao Guo

Abstract: Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video di… ▽ More Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications. △ Less

Submitted 4 June, 2025; originally announced June 2025.

Carbon-rich dust injected into the interstellar medium by Galactic WC binaries survives for hundreds of years

Authors: Noel D. Richardson, Micaela Henson, Emma P. Lieb, Corey Kehl, Ryan M. Lau, Peredur M. Williams, Michael F. Corcoran, J. R. Callingham, André-Nicolas Chené, Theodore R. Gull, Kenji Hamaguchi, Yinuo Han, Matthew J. Hankins, Grant M. Hill, Jennifer L. Hoffman, Jonathan Mackey, Anthony F. J. Moffat, Benjamin J. S. Pope, Pragati Pradhan, Christopher M. P. Russell, Andreas A. C. Sander, Nicole St-Louis, Ian R. Stevens, Peter Tuthill, Gerd Weigelt , et al. (1 additional authors not shown)

Abstract: Some carbon-rich Wolf-Rayet stars (WC stars) show an infrared excess from dust emission. Dust forms in the collision of the WC wind with a companion star's wind. As this dust is carried towards the ISM at close to the WCd wind speed and the binary continues through its orbit, a spiral structure forms around the system. The shape depends on the orbital eccentricity and period, as well as stellar pa… ▽ More Some carbon-rich Wolf-Rayet stars (WC stars) show an infrared excess from dust emission. Dust forms in the collision of the WC wind with a companion star's wind. As this dust is carried towards the ISM at close to the WCd wind speed and the binary continues through its orbit, a spiral structure forms around the system. The shape depends on the orbital eccentricity and period, as well as stellar parameters like mass-loss rates and terminal wind speeds. Imaging of the WCd binary WR 140 with JWST/MIRI revealed 17 concentric dust shells surrounding the binary. We present new JWST imaging of four additional WCd systems (WR 48a, WR 112, WR 125, and WR 137) that were imaged in 2024. In this analysis, we show that the dust is long-lived, detected with an age of at least 130 years, but more than 300 years in some systems. Longer duration measurements are limited by sensitivity. Regular spacing of dust features confirms the periodic nature of dust formation, consistent with a connection to binary motion. We use these images to estimate the proper motion of the dust, finding the dust to propagate out to the interstellar medium with motion comparable to the wind speed of the WC stars. In addition to these results, we observe unusual structures around WR 48a, which could represent dusty clumps shaped by photoevaporation and wind ablation like young proplyd objects. These results demonstrate that WC dust is indeed long-lived and should be accounted for in galactic dust budgets. △ Less

Submitted 29 May, 2025; v1 submitted 16 May, 2025; originally announced May 2025.

Comments: accepted to ApJ

WaterDrum: Watermarking for Data-centric Unlearning Metric

Authors: Xinyang Lu, Xinyuan Niu, Gregory Kang Ruey Lau, Bui Thi Cam Nhung, Rachael Hwee Ling Sim, Fanyu Wen, Chuan-Sheng Foo, See-Kiong Ng, Bryan Kian Hsiang Low

Abstract: Large language model (LLM) unlearning is critical in real-world applications where it is necessary to efficiently remove the influence of private, copyrighted, or harmful data from some users. However, existing utility-centric unlearning metrics (based on model utility) may fail to accurately evaluate the extent of unlearning in realistic settings such as when (a) the forget and retain set have se… ▽ More Large language model (LLM) unlearning is critical in real-world applications where it is necessary to efficiently remove the influence of private, copyrighted, or harmful data from some users. However, existing utility-centric unlearning metrics (based on model utility) may fail to accurately evaluate the extent of unlearning in realistic settings such as when (a) the forget and retain set have semantically similar content, (b) retraining the model from scratch on the retain set is impractical, and/or (c) the model owner can improve the unlearning metric without directly performing unlearning on the LLM. This paper presents the first data-centric unlearning metric for LLMs called WaterDrum that exploits robust text watermarking for overcoming these limitations. We also introduce new benchmark datasets for LLM unlearning that contain varying levels of similar data points and can be used to rigorously evaluate unlearning algorithms using WaterDrum. Our code is available at https://github.com/lululu008/WaterDrum and our new benchmark datasets are released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax. △ Less

Submitted 8 May, 2025; originally announced May 2025.

Very Late-Time JWST and Keck Spectra of the Oxygen-Rich Supernova 1995N

Authors: Geoffrey C. Clayton, R. Wesson, Ori D. Fox, Melissa Shahbandeh, Alexei V. Filippenko, Bryony Nickson, Michael Engesser, Schuyler D. Van Dyk, WeiKang Zheng, Thomas G. Brink, Yi Yang, Tea Temim, Nathan Smith, Jennifer Andrews, Chris Ashall, Ilse De Looze, James M. Derkacy, Luc Dessart, Michael Dulude, Eli Dwek, Ryan J. Foley, Suvi Gezari, Sebastian Gomez, Shireen Gonzaga, Siva Indukuri , et al. (21 additional authors not shown)

Abstract: We present new {\it JWST}/MIRI MRS and Keck spectra of SN 1995N obtained in 2022--2023, more than 10,000 days after the supernova (SN) explosion. These spectra are among the latest direct detections of a core-collapse SN, both through emission lines in the optical and thermal continuum from infrared dust emission. The new infrared data show that dust heating from radiation produced by the ejecta i… ▽ More We present new {\it JWST}/MIRI MRS and Keck spectra of SN 1995N obtained in 2022--2023, more than 10,000 days after the supernova (SN) explosion. These spectra are among the latest direct detections of a core-collapse SN, both through emission lines in the optical and thermal continuum from infrared dust emission. The new infrared data show that dust heating from radiation produced by the ejecta interacting with circumstellar matter is still present, but greatly reduced from when SN 1995N was observed by the {\it Spitzer Space Telescope} and {\it WISE} in 2009/2010 and 2018, when the dust mass was estimated to be 0.4 M(Sun). New radiative-transfer modeling suggests that the dust mass and grain size may have increased between 2010 and 2023. The new data can alternatively be well fit with a dust mass of 0.4 M(Sun) and a much reduced heating source luminosity. The new late-time spectra show unusually strong oxygen forbidden lines, stronger than the H-alpha emission. This indicates that SN 1995N may have exploded as a stripped-envelope SN which then interacted with a massive H-rich circumstellar shell, changing it from intrinsically Type Ib/c to Type IIn. The late-time spectrum results when the reverse shock begins to excite the inner H-poor, O-rich ejecta. This change in the spectrum is rarely seen, but marks the start of the transition from SN to SN remnant. △ Less

Submitted 2 May, 2025; originally announced May 2025.

Comments: 14 pages, 8 figures, ApJ Submitted

Large Cold Dust Reservoir Revealed in Transitional SN Ib 2014C by James Webb Space Telescope Mid-Infrared Spectroscopy

Authors: Samaporn Tinyanont, Ori D. Fox, Melissa Shahbandeh, Tea Temim, Robert Williams, Kittipong Wangnok, Armin Rest, Ryan M. Lau, Keiichi Maeda, Jacob E. Jencson, Katie Auchettl, Alexei V. Filippenko, Conor Larison, Christopher Ashall, Thomas Brink, Kyle W. Davis, Luc Dessart, Ryan J. Foley, Lluís Galbany, Matthew Grayling, Joel Johansson, Mansi M. Kasliwal, Zachary G. Lane, Natalie LeBaron, Dan Milisavljevic , et al. (10 additional authors not shown)

Abstract: Supernova (SN) 2014C is a rare transitional event that exploded as a hydrogen-poor, helium-rich Type Ib SN and subsequently interacted with a hydrogen-rich circumstellar medium (CSM) a few months post-explosion. This unique interacting object provides an opportunity to probe the mass-loss history of a stripped-envelope SN progenitor. Using the James Webb Space Telescope (JWST), we observed SN 2014… ▽ More Supernova (SN) 2014C is a rare transitional event that exploded as a hydrogen-poor, helium-rich Type Ib SN and subsequently interacted with a hydrogen-rich circumstellar medium (CSM) a few months post-explosion. This unique interacting object provides an opportunity to probe the mass-loss history of a stripped-envelope SN progenitor. Using the James Webb Space Telescope (JWST), we observed SN 2014C with the Mid-Infrared Instrument Medium Resolution Spectrometer at 3477 days post-explosion (rest frame), and the Near-Infrared Spectrograph Integral Field Unit at 3568 days post-explosion, covering 1.7 to 25 $μ$m. The bolometric luminosity indicates that the SN is still interacting with the same CSM that was observed with the Spitzer Space Telescope 40--1920 days post-explosion. JWST spectra and near-contemporaneous optical and near-infrared spectra show strong [Ne II] 12.831 $μ$m, He 1.083 $μ$m, H$α$, and forbidden oxygen ([O I] $λ$$λ$6300, 6364, [O II] $λ$$λ$7319, 7330, and [O III] $λ$$λ$4959, 5007) emission lines with asymmetric profiles, suggesting a highly asymmetric CSM. The mid-IR continuum can be explained by ~$0.036 \ M_\odot$ of carbonaceous dust at ~300 K and ~0.043 $M_\odot$ of silicate dust at ~200 K. The observed dust mass has increased tenfold since the last Spitzer observation 4 yr ago, with evidence suggesting that new grains have condensed in the cold dense shell between the forward and reverse shocks. This dust mass places SN 2014C among the dustiest SNe in the mid-IR and supports the emerging observational trend that SN explosions produce enough dust to explain the observed dust mass at high redshifts. △ Less

Submitted 4 June, 2025; v1 submitted 18 April, 2025; originally announced April 2025.

Comments: Published in ApJ

Nonlocal effects on Thermal Transport in MagLIF-Relevant Gaspipes on NIF

Authors: R. Y. Lau, D. J. Strozzi, M. Sherlock, M. Weis, A. S. Joglekar, W. A. Farmer, Y. Shi, J. R. Cary

Abstract: We present simulations of heat flow relevant to gaspipe experiments on the National Ignition Facility (NIF) to investigate kinetic effects on transport phenomena. D2 and neopentane (C5H12) filled targets are used to study the laser preheat stage of a MagLIF scheme where anaxial magnetic field is sometimes applied to the target. Simulations were done with the radiation-MHD code HYDRA with a collisi… ▽ More We present simulations of heat flow relevant to gaspipe experiments on the National Ignition Facility (NIF) to investigate kinetic effects on transport phenomena. D2 and neopentane (C5H12) filled targets are used to study the laser preheat stage of a MagLIF scheme where anaxial magnetic field is sometimes applied to the target. Simulations were done with the radiation-MHD code HYDRA with a collision-dominated fluid model and the Schurtz nonlocal electron thermal conduction model. Using the Schurtz model to evolve the electron temperature increased the heat front propagation of neopentane gas targets compared to a local model by limiting radial heat flow. This increases electron temperature near the axis, which decreases laser absorption. We find the effect of heat flow models on temperature profiles and laser propagation is modest. Beyond the Schurtz model, we utilize HYDRA to initialize plasma conditions for the Vlasov Fokker-Planck K2 code. We run K2 until a quasi-steady state is reached and examine the impact of kinetic effects on heat transport. Although axial heat flow is well predicted by fluid models, the fluid model consistently over predicts radial heat flow up to 150% in regions with the largest temperature gradient of D2 filled gaspipes. On the other hand, the Schurtz nonlocal electron conduction model is found to be adequate for capturing kinetic heat flow in gaspipes. △ Less

Submitted 12 April, 2025; originally announced April 2025.

Comments: 12 pages, 10 figures

Crust composition and the Shallow Heat Source in KS 1731-260

Authors: R. Jain, E. F. Brown, H. Schatz, A. V. Afanasjev, M. Beard, L. R. Gasques, J. Grace, A. Heger, G. W. Hitt, W. R. Hix, R. Lau, W. -J. Ong, M. Wiescher, Y. Xu

Abstract: The presence of a shallow heat source of unknown origin in accreting neutron star crusts has been inferred by analyzing their cooling behavior in quiescence. To investigate a diverse bursting history for KS 1731-260 during accretion outbursts, we use realistic crust compositions and nuclear heating and cooling sources from detailed nuclear reaction network calculations to interpret observed coolin… ▽ More The presence of a shallow heat source of unknown origin in accreting neutron star crusts has been inferred by analyzing their cooling behavior in quiescence. To investigate a diverse bursting history for KS 1731-260 during accretion outbursts, we use realistic crust compositions and nuclear heating and cooling sources from detailed nuclear reaction network calculations to interpret observed cooling curves. We find that the required strength of the shallow heat source is reduced by more than a factor of 3 compared to previous analysis, and obtain constraints on the most likely dominant surface burning modes of KS 1731-260 over its history. Our analysis suggests an impure nuclear pasta layer in the inner crust, though future observations will provide more stringent constraints. △ Less

Submitted 11 April, 2025; originally announced April 2025.

Revealing a main-sequence star that consumed a planet with JWST

Authors: Ryan M. Lau, Jacob E. Jencson, Colette Salyk, Kishalay De, Ori D. Fox, Matthew J. Hankins, Mansi M. Kasliwal, Charles D. Keyes, Morgan Macleod, Michael E. Ressler, Sam Rose

Abstract: The subluminous red nova (SLRN) ZTF SLRN-2020 is the most compelling direct detection of a planet being consumed by its host star, a scenario known as a planetary engulfment event. We present JWST spectroscopy of ZTF SLRN-2020 taken +830 d after its optical emission peak using the NIRSpec fixed-slit $3-5$ $μ$m high-resolution grating and the MIRI $5-12$ $μ$m low-resolution spectrometer. NIRSpec re… ▽ More The subluminous red nova (SLRN) ZTF SLRN-2020 is the most compelling direct detection of a planet being consumed by its host star, a scenario known as a planetary engulfment event. We present JWST spectroscopy of ZTF SLRN-2020 taken +830 d after its optical emission peak using the NIRSpec fixed-slit $3-5$ $μ$m high-resolution grating and the MIRI $5-12$ $μ$m low-resolution spectrometer. NIRSpec reveals the $^{12}$CO fundamental band ($ν=1-0$) in emission at $\sim4.7$ $μ$m, Brackett-$α$ emission, and the potential detection of PH$_3$ in emission at $\sim4.3$ $μ$m. The JWST spectra are consistent with the claim that ZTF SLRN-2020 arose from a planetary engulfment event. We utilize DUSTY to model the late-time $\sim1-12$ $μ$m spectral energy distribution (SED) of ZTF SLRN-2020, where the best-fit parameters indicate the presence of warm, $720^{+80}_{-50}$ K, circumstellar dust with a total dust mass of Log$\left(\frac{M_\mathrm{d}}{\mathrm{M}_\odot}\right)=-10.61^{+0.08}_{-0.16}$ M$_\odot$. We also fit a DUSTY model to archival photometry taken +320 d after peak that suggested the presence of a cooler, T$_\mathrm{d}=280^{+450}_{-20}$ K, and more massive, Log$\left(\frac{M_\mathrm{d}}{\mathrm{M}_\odot}\right)=-5.89^{+0.29}_{-3.21}$, circumstellar dust component. Assuming the cool component originates from the ZTF SLRN-2020 ejecta, we interpret the warm component as fallback from the ejecta. From the late-time SED model we measure a luminosity of L$_* = 0.29^{+0.03}_{-0.06}$ L$_\odot$ for the remnant host star, which is consistent with a $\sim0.7$ M$_\odot$ K-type star that should not yet have evolved off the main sequence. If ZTF SLRN-2020 was not triggered by stellar evolution, we suggest that the planetary engulfment was due to orbital decay from tidal interactions between the planet and the host star. △ Less

Submitted 9 April, 2025; originally announced April 2025.

Comments: Published in ApJ on Apr 10, 2025; 22 pages, 9 figures, 3 tables

JWST/MIRI detects the dusty SN1993J about 30 years after explosion

Authors: Tamás Szalai, Szanna Zsíros, Jacob Jencson, Ori D. Fox, Melissa Shahbandeh, Arkaprabha Sarangi, Tea Temim, Ilse De Looze, Nathan Smith, Alexei V. Filippenko, Schuyler D. Van Dyk, Jennifer Andrews, Chris Ashall, Geoffrey C. Clayton, Luc Dessart, Michael Dulude, Eli Dwek, Sebastian Gomez, Joel Johansson, Dan Milisavljevic, Justin Pierel, Armin Rest, Samaporn Tinyanont, Thomas G. Brink, Kishalay De , et al. (15 additional authors not shown)

Abstract: Core-collapse supernovae (CCSNe) have long been considered to contribute significantly to the cosmic dust budget. New dust cools quickly and is therefore detectable at mid-infrared (mid-IR) wavelengths. However, before the era of the James Webb Space Telescope (JWST), direct observational evidence for dust condensation was found in only a handful of nearby CCSNe, and dust masses (~10… ▽ More Core-collapse supernovae (CCSNe) have long been considered to contribute significantly to the cosmic dust budget. New dust cools quickly and is therefore detectable at mid-infrared (mid-IR) wavelengths. However, before the era of the James Webb Space Telescope (JWST), direct observational evidence for dust condensation was found in only a handful of nearby CCSNe, and dust masses (~10$^{-2}-10^{-3} M_{\odot}$, generally limited to <5 yr and to >500K temperatures) have been 2-3 orders of magnitude smaller than either theoretical predictions or dust amounts found by far-IR/submm observations of Galactic SN remnants and in the very nearby SN 1987A. The combined angular resolution and mid-IR sensitivity of JWST finally allow us to reveal hidden cool (~100-200K) dust reservoirs in extragalactic SNe beyond SN 1987A. Our team received JWST/MIRI time for studying a larger sample of CCSNe to fill the currently existing gap in their dust formation histories. The first observed target of this program is the well-known Type IIb SN~1993J appeared in M81. We generated its spectral energy distribution (SED) from the current JWST/MIRI F770W, F1000W, F1500W, and F2100W fluxes. We fit single- and two-component silicate and carbonaceous dust models to the SED. We found that SN 1993J still contains a significant amount (~0.01 $M_{\odot}$) of dust ~30 yr after explosion. Comparing these results to those of the analysis of earlier {Spitzer Space Telescope data, we see a similar amount of dust now that was detected ~15-20 yr ago, but at a lower temperature. We also find residual background emission near the SN site (after point-spread-function subtraction on the JWST/MIRI images) that may plausibly be attributed to an IR echo from more distant interstellar dust grains heated by the SN shock-breakout luminosity or ongoing star formation in the local environment. △ Less

Submitted 17 March, 2025; originally announced March 2025.

Comments: 10 pages, 9 figures, 2 Tables; accepted for publication in A&A

Journal ref: A&A 697, A132 (2025)

Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

Authors: Youjun Zhao, Jiaying Lin, Rynson W. H. Lau

Abstract: Open-vocabulary 3D object detection (OV-3DOD) aims at localizing and classifying novel objects beyond closed sets. The recent success of vision-language models (VLMs) has demonstrated their remarkable capabilities to understand open vocabularies. Existing works that leverage VLMs for 3D object detection (3DOD) generally resort to representations that lose the rich scene context required for 3D per… ▽ More Open-vocabulary 3D object detection (OV-3DOD) aims at localizing and classifying novel objects beyond closed sets. The recent success of vision-language models (VLMs) has demonstrated their remarkable capabilities to understand open vocabularies. Existing works that leverage VLMs for 3D object detection (3DOD) generally resort to representations that lose the rich scene context required for 3D perception. To address this problem, we propose in this paper a hierarchical framework, named HCMA, to simultaneously learn local object and global scene information for OV-3DOD. Specifically, we first design a Hierarchical Data Integration (HDI) approach to obtain coarse-to-fine 3D-image-text data, which is fed into a VLM to extract object-centric knowledge. To facilitate the association of feature hierarchies, we then propose an Interactive Cross-Modal Alignment (ICMA) strategy to establish effective intra-level and inter-level feature connections. To better align features across different levels, we further propose an Object-Focusing Context Adjustment (OFCA) module to refine multi-level features by emphasizing object-related features. Extensive experiments demonstrate that the proposed method outperforms SOTA methods on the existing OV-3DOD benchmarks. It also achieves promising OV-3DOD results even without any 3D annotations. △ Less

Submitted 10 March, 2025; originally announced March 2025.

Comments: AAAI 2025 (Extented Version). Project Page: https://youjunzhao.github.io/HCMA/

PIED: Physics-Informed Experimental Design for Inverse Problems

Authors: Apivich Hemachandra, Gregory Kang Ruey Lau, See-Kiong Ng, Bryan Kian Hsiang Low

Abstract: In many science and engineering settings, system dynamics are characterized by governing PDEs, and a major challenge is to solve inverse problems (IPs) where unknown PDE parameters are inferred based on observational data gathered under limited budget. Due to the high costs of setting up and running experiments, experimental design (ED) is often done with the help of PDE simulations to optimize fo… ▽ More In many science and engineering settings, system dynamics are characterized by governing PDEs, and a major challenge is to solve inverse problems (IPs) where unknown PDE parameters are inferred based on observational data gathered under limited budget. Due to the high costs of setting up and running experiments, experimental design (ED) is often done with the help of PDE simulations to optimize for the most informative design parameters to solve such IPs, prior to actual data collection. This process of optimizing design parameters is especially critical when the budget and other practical constraints make it infeasible to adjust the design parameters between trials during the experiments. However, existing experimental design (ED) methods tend to require sequential and frequent design parameter adjustments between trials. Furthermore, they also have significant computational bottlenecks due to the need for complex numerical simulations for PDEs, and do not exploit the advantages provided by physics informed neural networks (PINNs), such as its meshless solutions, differentiability, and amortized training. This work presents PIED, the first ED framework that makes use of PINNs in a fully differentiable architecture to perform continuous optimization of design parameters for IPs for one-shot deployments. PIED overcomes existing methods' computational bottlenecks through parallelized computation and meta-learning of PINN parameter initialization, and proposes novel methods to effectively take into account PINN training dynamics in optimizing the ED parameters. Through experiments based on noisy simulated data and even real world experimental data, we empirically show that given limited observation budget, PIED significantly outperforms existing ED methods in solving IPs, including challenging settings where the inverse parameters are unknown functions rather than just finite-dimensional. △ Less

Submitted 10 March, 2025; originally announced March 2025.

Comments: Accepted to 13th International Conference on Learning Representations (ICLR 2025), 31 pages

JWST/MIRI Study of the Enigmatic Mid-Infrared Rings in the Planetary Nebula NGC 1514

Authors: Michael E. Ressler, Alba Aller, David Jones, Ryan M. Lau, Luis F. Miranda, Karen Willacy

Abstract: While NGC 1514 is an elliptical, but complex, planetary nebula at optical wavelengths, it was discovered to have a pair of infrared-bright, axisymmetric rings contained within its faint outer shell during the course of the WISE all-sky survey. We have obtained JWST mid-infrared imaging and spectroscopy of the nebula through the use of simultaneous observations with the MIRI Imager and Medium Resol… ▽ More While NGC 1514 is an elliptical, but complex, planetary nebula at optical wavelengths, it was discovered to have a pair of infrared-bright, axisymmetric rings contained within its faint outer shell during the course of the WISE all-sky survey. We have obtained JWST mid-infrared imaging and spectroscopy of the nebula through the use of simultaneous observations with the MIRI Imager and Medium Resolution Spectrometer, selecting the F770W, F1280W, and F2550W filters to match each of the MRS's three grating positions. These observations show that the rings are clearly resolved and relatively distinct structures, with both filamentary and clumpy detail throughout. There is also cloud-like material that has a turbulent appearance in the interior of the rings, particularly at the longest wavelengths, and faint ejecta-like structures just outside the ring boundaries. Despite their brightness, the emission from the rings within the three imager passbands is shown to be dominated by thermal emission from very small grains, not line emission from atomic hydrogen or forbidden atomic lines, shocked molecular hydrogen, or PAHs. The doppler velocities derived from the two brightest emission lines in the rings, however, suggest that the material from which the rings were formed was ejected during an early period of very heavy mass loss from the PN progenitor, then shaped by asymmetrical fast winds from the central binary pair. △ Less

Submitted 21 March, 2025; v1 submitted 28 February, 2025; originally announced February 2025.

Comments: Accepted for publication in the Astronomical Journal, 21 pages, 18 figures, 7 tables. V2 fixes typos in an ORCID, an institutional address, and a contract identifier

Cryoscope: A Cryogenic Infrared Survey Telescope in Antarctica

Authors: Mansi M. Kasliwal, Nicholas Earley, Roger Smith, Tristan Guillot, Tony Travouillon, Jason Fucik, Lyu Abe, Timothee Greffe, Abdelkrim Agabi, Michael C. B. Ashley, Amaury H. M. J. Triaud, Samaporn Tinyanont, Sarah Antier, Philippe Bendjoya, Rohan Bhattarai, Rob Bertz, James Brugger, Artem Burdanov, Ilaria Caiazzo, Benoit Carry, Luca Casagrande, Brad Cenko, Jeff Cooke, Kishalay De, Richard Dekany , et al. (36 additional authors not shown)

Abstract: We present Cryoscope--a new 50 deg$^2$ field-of-view, 1.2 m aperture, $K_{dark}$ survey telescope to be located at Dome C, Antarctica. Cryoscope has an innovative optical-thermal design wherein the entire telescope is cryogenically cooled. Cryoscope also explores new detector technology to cost-effectively tile the full focal plane. Leveraging the dark Antarctic sky and minimizing telescope therma… ▽ More We present Cryoscope--a new 50 deg$^2$ field-of-view, 1.2 m aperture, $K_{dark}$ survey telescope to be located at Dome C, Antarctica. Cryoscope has an innovative optical-thermal design wherein the entire telescope is cryogenically cooled. Cryoscope also explores new detector technology to cost-effectively tile the full focal plane. Leveraging the dark Antarctic sky and minimizing telescope thermal emission, Cryoscope achieves unprecedented deep, wide, fast and red observations, matching and exceeding volumetric survey speeds from the Ultraviolet Explorer, Vera Rubin Observatory, Nancy Grace Roman Space Telescope, SPHEREx, and NEO Surveyor. By providing coverage beyond wavelengths of 2 $μ$m, we aim to create the most comprehensive dynamic movie of the most obscured reaches of the Universe. Cryoscope will be a dedicated discovery engine for electromagnetic emission from coalescing compact binaries, Earth-like exoplanets orbiting cold stars, and multiple facets of time-domain, stellar and solar system science. In this paper, we describe the scientific drivers and technical innovations for this new discovery engine operating in the $K_{dark}$ passband, why we choose to deploy it in Antarctica, and the status of a fifth-scale prototype designed as a Pathfinder to retire technological risks prior to full-scale implementation. We plan to deploy the Cryoscope Pathfinder to Dome C in December 2026 and the full-scale telescope by 2030. △ Less

Submitted 21 March, 2025; v1 submitted 10 February, 2025; originally announced February 2025.

Comments: 40 pages, 19 figures, 4 tables; accepted for publication in PASP on 2025-03-21

Dynamic Imprints of Colliding-wind Dust Formation from WR140

Authors: Emma P. Lieb, Ryan M. Lau, Jennifer L. Hoffman, Michael F. Corcoran, Macarena Garcia Marin, Theodore R. Gull, Kenji Hamaguchi, Yinuo Han, Matthew J. Hankins, Olivia C. Jones, Thomas I. Madura, Sergey V. Marchenko, Hideo Matsuhara, Florentin Millour, Anthony F. J. Moffat, Mark R. Morris, Patrick W. Morris, Takashi Onaka, Marshall D. Perrin, Armin Rest, Noel Richardson, Christopher M. P. Russell, Joel Sanchez-Bermudez, Anthony Soulain, Peter Tuthill , et al. (2 additional authors not shown)

Abstract: Carbon-rich Wolf-Rayet binaries are a prominent source of carbonaceous dust that contribute to the dust budget of galaxies. The "textbook" example of an episodic dust producing WR binary, WR140 (HD193793), provides us with an ideal laboratory for investigating the dust physics and kinematics in an extreme environment. This study is among the first to utilize two separate JWST observations, from Cy… ▽ More Carbon-rich Wolf-Rayet binaries are a prominent source of carbonaceous dust that contribute to the dust budget of galaxies. The "textbook" example of an episodic dust producing WR binary, WR140 (HD193793), provides us with an ideal laboratory for investigating the dust physics and kinematics in an extreme environment. This study is among the first to utilize two separate JWST observations, from Cycle 1 ERS (July 2022) and Cycle 2 (Sept. 2023), to measure WR140's dust kinematics and confirm its morphology. To measure the proper motions and projected velocities of the dust shells, we performed a novel PSF subtraction to reduce the effects of the bright diffraction spikes and carefully aligned the Cycle 2 to the Cycle 1 images. At 7.7 $μ$m, through the bright feature common to 16 dust shells (C1), we find an average dust shell proper motion of $390\pm29$ mas yr$^{-1}$, which equates to a projected velocity of $2714\pm188$ km s$^{-1}$ at a distance of 1.64 kpc. Our measured speeds are constant across all visible shells and consistent with previously reported dust expansion velocities. Our observations not only prove that these dusty shells are astrophysical (i.e., not associated with any PSF artifact) and originate from WR140, but also confirm the "clumpy" morphology of the dust shells, in which identifiable substructures within certain shells persist for at least 14 months from one cycle to the next. These results support the hypothesis that clumping in the wind collision region is required for dust production in WR binaries. △ Less

Submitted 4 February, 2025; originally announced February 2025.

Journal ref: The ApJL Vol. 979 (2025) Num. 1

DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks

Authors: Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo, Bryan Kian Hsiang Low

Abstract: The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often unknown (e.g., conversations between an LLM and a user are end-to-end encrypted). Hence, it is unclear what data are relevant for fine-tuning the LLM to maximize its performance on the specific unseen evalua… ▽ More The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often unknown (e.g., conversations between an LLM and a user are end-to-end encrypted). Hence, it is unclear what data are relevant for fine-tuning the LLM to maximize its performance on the specific unseen evaluation task. Instead, one can only deploy the LLM on the unseen task to gather multiple rounds of feedback on how well the model performs (e.g., user ratings). This novel setting offers a refreshing perspective towards optimizing training data mixtures via feedback from an unseen evaluation task, which prior data mixing and selection works do not consider. Our paper presents DUET, a novel global-to-local algorithm that interleaves influence function as a data selection method with Bayesian optimization to optimize data mixture via feedback from a specific unseen evaluation task. By analyzing DUET's cumulative regret, we theoretically show that DUET converges to the optimal training data mixture for an unseen task even without any data knowledge of the task. Finally, our experiments across a variety of language tasks demonstrate that DUET outperforms existing data selection and mixing methods in the unseen-task setting. △ Less

Submitted 18 May, 2025; v1 submitted 31 January, 2025; originally announced February 2025.

Pre-trained Audio Transformer as a Foundational AI Tool for Gravitational Waves

Authors: Chayan Chatterjee, Abigail Petulante, Karan Jani, Jesse Spencer-Smith, Yang Hu, Roy Lau, Haowei Fu, Trang Hoang, Stephen Chong Zhao, Suyash Deshmukh

Abstract: As gravitational wave detectors become more advanced and sensitive, the number of signals recorded by Advanced LIGO and Virgo from merging compact objects is expected to rise dramatically. This surge in detection rates necessitates the development of adaptable, scalable, and efficient tools capable of addressing a wide range of tasks in gravitational wave astronomy. Foundational AI models present… ▽ More As gravitational wave detectors become more advanced and sensitive, the number of signals recorded by Advanced LIGO and Virgo from merging compact objects is expected to rise dramatically. This surge in detection rates necessitates the development of adaptable, scalable, and efficient tools capable of addressing a wide range of tasks in gravitational wave astronomy. Foundational AI models present a transformative opportunity in this context by providing a unified framework that can be fine tuned for diverse applications while leveraging the power of large scale pre training. In this work, we explore how advanced transformer models, specifically Whisper by OpenAI, can be adapted as a foundational model for gravitational wave data analysis. By fine tuning the encoder model of Whisper, originally trained on extensive audio data, and combining it with neural networks for specialized tasks, we achieve reliable results in detecting astrophysical signals and classifying transient noise artifacts or glitches. This represents the first application of open source transformer models, pre trained on unrelated tasks, for gravitational wave research, demonstrating their potential to enable versatile and efficient data analysis in the era of rapidly increasing detection rates. △ Less

Submitted 30 December, 2024; originally announced December 2024.

Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

Authors: Gregory Kang Ruey Lau, Wenyang Hu, Diwen Liu, Jizhuo Chen, See-Kiong Ng, Bryan Kian Hsiang Low

Abstract: Large Language Models still encounter substantial challenges in reasoning tasks, especially for smaller models, which many users may be restricted to due to resource constraints (e.g. GPU memory restrictions). Inference-time methods to boost LLM performance, such as prompting methods to invoke certain reasoning pathways in responses, have been shown effective in past works, though they largely rel… ▽ More Large Language Models still encounter substantial challenges in reasoning tasks, especially for smaller models, which many users may be restricted to due to resource constraints (e.g. GPU memory restrictions). Inference-time methods to boost LLM performance, such as prompting methods to invoke certain reasoning pathways in responses, have been shown effective in past works, though they largely rely on sequential queries. The ensemble method, which consists of multiple constituent models running in parallel, is a promising approach to achieving better inference-time performance, especially given recent developments that enabled significant speed-ups in LLM batch inference. In this work, we propose a novel, training-free LLM ensemble framework where a single LLM model is fed an optimized, diverse set of prompts in parallel, effectively producing an ensemble at inference time to achieve performance improvement in reasoning tasks. We empirically demonstrate that our method leads to significant gains on math reasoning tasks, e.g., on MATH, where our ensemble consisting of a few small models (e.g., three Qwen2-MATH-1.5B-it models) can outperform a larger model (e.g., Qwen2-MATH-7B-it). △ Less

Submitted 12 December, 2024; originally announced December 2024.

Comments: Accepted to NeurIPS 2024 Workshop on Foundation Model Interventions (MINT)

Do Multimodal Large Language Models See Like Humans?

Authors: Jiaying Lin, Shuquan Ye, Rynson W. H. Lau

Abstract: Multimodal Large Language Models (MLLMs) have achieved impressive results on various vision tasks, leveraging recent advancements in large language models. However, a critical question remains unaddressed: do MLLMs perceive visual information similarly to humans? Current benchmarks lack the ability to evaluate MLLMs from this perspective. To address this challenge, we introduce HVSBench, a large-s… ▽ More Multimodal Large Language Models (MLLMs) have achieved impressive results on various vision tasks, leveraging recent advancements in large language models. However, a critical question remains unaddressed: do MLLMs perceive visual information similarly to humans? Current benchmarks lack the ability to evaluate MLLMs from this perspective. To address this challenge, we introduce HVSBench, a large-scale benchmark designed to assess the alignment between MLLMs and the human visual system (HVS) on fundamental vision tasks that mirror human vision. HVSBench curated over 85K multimodal samples, spanning 13 categories and 5 fields in HVS, including Prominence, Subitizing, Prioritizing, Free-Viewing, and Searching. Extensive experiments demonstrate the effectiveness of our benchmark in providing a comprehensive evaluation of MLLMs. Specifically, we evaluate 13 MLLMs, revealing that even the best models show significant room for improvement, with most achieving only moderate results. Our experiments reveal that HVSBench presents a new and significant challenge for cutting-edge MLLMs. Diverse human participants attained strong performance, significantly outperforming MLLMs, which further underscores the benchmark's high quality. We believe that HVSBench will facilitate research on human-aligned and explainable MLLMs, marking a key step in understanding how MLLMs perceive and process visual information. △ Less

Submitted 27 March, 2025; v1 submitted 12 December, 2024; originally announced December 2024.

Comments: Project page: https://jiaying.link/HVSBench/

Revisiting the Integration of Convolution and Attention for Vision Backbone

Authors: Lei Zhu, Xinjiang Wang, Wayne Zhang, Rynson W. H. Lau

Abstract: Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such… ▽ More Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{https://github.com/rayleizhu/GLMix}. △ Less

Submitted 21 November, 2024; originally announced November 2024.

Comments: NeurIPS 2024

LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes

Authors: Zefan Qu, Ke Xu, Gerhard Petrus Hancke, Rynson W. H. Lau

Abstract: Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additiona… ▽ More Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additional image-based enhancement methods does not work as these degradation factors are highly coupled. We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process. To this end, we propose in this paper a novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images. The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, respectively. Specifically, LuSh-NeRF includes a novel Scene-Noise Decomposition (SND) module for decoupling the noise from the scene representation and a novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information. To facilitate training and evaluations, we construct a new dataset containing both synthetic and real images. Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: https://github.com/quzefan/LuSh-NeRF. △ Less

Submitted 11 November, 2024; originally announced November 2024.

Comments: Accepted by NeurIPS 2024

The disappearance of a massive star marking the birth of a black hole in M31

Authors: Kishalay De, Morgan MacLeod, Jacob E. Jencson, Elizabeth Lovegrove, Andrea Antoni, Erin Kara, Mansi M. Kasliwal, Ryan M. Lau, Abraham Loeb, Megan Masterson, Aaron M. Meisner, Christos Panagiotou, Eliot Quataert, Robert Simcoe

Abstract: Stellar mass black holes are formed from the terminal collapse of massive stars if the ensuing neutrino shock is unable to eject the stellar envelope. Direct observations of black hole formation remain inconclusive. We report observations of M31-2014-DS1, a massive, hydrogen-depleted supergiant in the Andromeda galaxy identified via a mid-infrared brightening in 2014. Its total luminosity remained… ▽ More Stellar mass black holes are formed from the terminal collapse of massive stars if the ensuing neutrino shock is unable to eject the stellar envelope. Direct observations of black hole formation remain inconclusive. We report observations of M31-2014-DS1, a massive, hydrogen-depleted supergiant in the Andromeda galaxy identified via a mid-infrared brightening in 2014. Its total luminosity remained nearly constant for the subsequent thousand days, before fading dramatically over the next thousand days by $\gtrsim 10\times$ and $\gtrsim 10^4\times$ in total and visible light, respectively. Together with the lack of a detected optical outburst, the observations are explained by the fallback of the stellar envelope into a newly formed black hole, moderated by the injection of a $\sim 10^{48}$ erg shock. Unifying these observations with a candidate in NGC 6946, we present a concordant picture for the birth of stellar mass black holes from stripped massive stars. △ Less

Submitted 18 October, 2024; originally announced October 2024.

Comments: Submitted for review

Visual Orbits of Wolf-Rayet Stars I: The Orbit of the dust-producing Wolf-Rayet binary WR\,137 measured with the CHARA Array

Authors: Noel D. Richardson, Gail H. Schaefer, Jan J. Eldridge, Rebecca Spejcher, Amanda Holdsworth, Ryan M. Lau, John D. Monnier, Anthony F. J. Moffat, Gerd Weigelt, Peredur M. Williams, Stefan Kraus, Jean-Baptiste Le Bouquin, Narsireddy Anugu, Sorabh Chhabra, Isabelle Codron, Jacob Ennis, Tyler Gardner, Mayra Gutierrez, Noura Ibrahim, Aaron Labdon, Cyprien Lanthermann, Benjamin R. Setterholm

Abstract: Classical Wolf-Rayet stars are the descendants of massive OB stars that have lost their hydrogen envelopes and are burning helium in their cores prior to exploding as type Ib/c supernovae. The mechanisms for losing their hydrogen envelopes are either through binary interactions or through strong stellar winds potentially coupled with episodic mass-loss. Amongst the bright classical WR stars, the b… ▽ More Classical Wolf-Rayet stars are the descendants of massive OB stars that have lost their hydrogen envelopes and are burning helium in their cores prior to exploding as type Ib/c supernovae. The mechanisms for losing their hydrogen envelopes are either through binary interactions or through strong stellar winds potentially coupled with episodic mass-loss. Amongst the bright classical WR stars, the binary system WR\,137 (HD\,192641; WC7d + O9e) is the subject of this paper. This binary is known to have a 13-year period and produces dust near periastron. Here we report on interferometry with the CHARA Array collected over a decade of time and providing the first visual orbit for the system. We combine these astrometric measurements with archival radial velocities to measure masses of the stars of $M_{\rm WR} = 9.5\pm3.4 M_\odot$ and $M_{\rm O} = 17.3\pm 1.9 M_\odot$ when we use the most recent \textit{Gaia} distance. These results are then compared to predicted dust distribution using these orbital elements, which match the observed imaging from \textit{JWST} as discussed recently by Lau et al. Furthermore, we compare the system to the BPASS models, finding that the WR star likely formed through stellar winds and not through binary interactions. However, the companion O star did likely accrete some material from the WR's mass-loss to provide the rotation seen today that drives its status as an Oe star. △ Less

Submitted 11 October, 2024; originally announced October 2024.

Comments: Accepted to ApJ

JWST/MIRI Observations of Newly Formed Dust in the Cold, Dense Shell of the Type IIn SN 2005ip

Authors: Melissa Shahbandeh, Ori D. Fox, Tea Temim, Eli Dwek, Arkaprabha Sarangi, Nathan Smith, Luc Dessart, Bryony Nickson, Michael Engesser, Alexei V. Filippenko, Thomas G. Brink, Weikang Zheng, Tamás Szalai, Joel Johansson, Armin Rest, Schuyler D. Van Dyk, Jennifer Andrews, Chris Ashall, Geoffrey C. Clayton, Ilse De Looze, James M. Derkacy, Michael Dulude, Ryan J. Foley, Suvi Gezari, Sebastian Gomez , et al. (20 additional authors not shown)

Abstract: Dust from core-collapse supernovae (CCSNe), specifically Type IIP SNe, has been suggested to be a significant source of the dust observed in high-redshift galaxies. CCSNe eject large amounts of newly formed heavy elements, which can condense into dust grains in the cooling ejecta. However, infrared (IR) observations of typical CCSNe generally measure dust masses that are too small to account for t… ▽ More Dust from core-collapse supernovae (CCSNe), specifically Type IIP SNe, has been suggested to be a significant source of the dust observed in high-redshift galaxies. CCSNe eject large amounts of newly formed heavy elements, which can condense into dust grains in the cooling ejecta. However, infrared (IR) observations of typical CCSNe generally measure dust masses that are too small to account for the dust production needed at high redshifts. Type IIn SNe, classified by their dense circumstellar medium (CSM), are also known to exhibit strong IR emission from warm dust, but the dust origin and heating mechanism have generally remained unconstrained because of limited observational capabilities in the mid-IR. Here, we present a JWST/MIRI Medium Resolution Spectrograph (MRS) spectrum of the Type IIn SN 2005ip nearly 17 years post-explosion. The Type IIn SN 2005ip is one of the longest-lasting and most well-studied SNe observed to date. Combined with a Spitzer mid-IR spectrum of SN 2005ip obtained in 2008, this data set provides a rare 15-year baseline, allowing for a unique investigation of the evolution of dust. The JWST spectrum shows a new high-mass dust component ($\gtrsim0.08$ M$_{\odot}$) that is not present in the earlier Spitzer spectrum. Our analysis shows dust likely formed over the past 15 years in the cold, dense shell (CDS), between the forward and reverse shocks. There is also a smaller mass of carbonaceous dust ($\gtrsim0.005$ M$_{\odot}$) in the ejecta. These observations provide new insights into the role of SN dust production, particularly within the CDS, and its potential contribution to the rapid dust enrichment of the early Universe. △ Less

Submitted 11 October, 2024; originally announced October 2024.

Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension

Authors: Zaiquan Yang, Yuhao Liu, Jiaying Lin, Gerhard Hancke, Rynson W. H. Lau

Abstract: This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (… ▽ More This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object. Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner. Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages. Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. Extensive experiments show that our method outperforms SOTA methods on three common benchmarks. △ Less

Submitted 4 December, 2024; v1 submitted 2 October, 2024; originally announced October 2024.

Comments: Accepted to NeurIPS2024

Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion

Authors: Zhenwei Wang, Tengfei Wang, Zexin He, Gerhard Hancke, Ziwei Liu, Rynson W. H. Lau

Abstract: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generaliz… ▽ More In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications. △ Less

Submitted 17 September, 2024; originally announced September 2024.

Comments: Project page: https://RAG-3D.github.io/

Possible anti-correlations between pulsation amplitudes and the disk growth of Be stars in giant-outbursting Be X-ray binaries

Authors: Masafumi Niwano, Michael M. Fausnaugh, Ryan M. Lau, Kishalay De, Roberto Soria, George R. Ricker, Roland Vanderspek, Michael C. B. Ashley, Nicholas Earley, Matthew J. Hankins, Mansi M. Kasliwal, Anna M. Moore, Jamie Soon, Tony Travouillon, Mahito Sasada, Ichiro Takahashi, Yoichi Yatsu, Nobuyuki Kawai

Abstract: The mechanism of X-ray outbursts in Be X-ray binaries remains a mystery, and understanding their circumstellar disks is crucial for a solution of the mass-transfer problem. In particular, it is important to identify the Be star activities (e.g., pulsations) that cause mass ejection and, hence, disk formation. Therefore, we investigated the relationship between optical flux oscillations and the inf… ▽ More The mechanism of X-ray outbursts in Be X-ray binaries remains a mystery, and understanding their circumstellar disks is crucial for a solution of the mass-transfer problem. In particular, it is important to identify the Be star activities (e.g., pulsations) that cause mass ejection and, hence, disk formation. Therefore, we investigated the relationship between optical flux oscillations and the infrared (IR) excess in a sample of five Be X-ray binaries. Applying the Lomb-Scargle technique to high-cadence optical light curves from the Transiting Exoplanet Survey Satellite (TESS), we detected several significant oscillation modes in the 3 to 24 hour period range for each source. We also measured the IR excess (a proxy for disk growth) of those five sources, using J-band light curves from Palomar Gattini-IR. In four of the five sources, we found anti-correlations between the IR excess and the amplitude of the main flux oscillation modes. This result is inconsistent with the conventional idea that non-radial pulsations drive mass ejections. We propose an alternative scenario where internal temperature variations in the Be star cause transitions between pulsation-active and mass-ejection-active states. △ Less

Submitted 14 September, 2024; originally announced September 2024.

Comments: 17 pages, 27 figures, 6 tables, accepted for publication in MNRAS

Revealing Potential Initial Mass Function variations with metallicity: JWST observations of young open clusters in a low-metallicity environment

Authors: Chikako Yasui, Natsuko Izumi, Masao Saito, Ryan M. Lau, Naoto Kobayashi, Michael E. Ressler

Abstract: We present the substellar mass function of star-forming clusters ($\simeq$0.1 Myr old) in a low-metallicity environment ($\simeq$$-$0.7 dex). We performed deep JWST/NIRCam and MIRI imaging of two star-forming clusters in Digel Cloud 2, a star-forming region in the Outer Galaxy ($R_G \gtrsim 15$ kpc). The very high sensitivity and spatial resolution of JWST enable us to resolve cluster members clea… ▽ More We present the substellar mass function of star-forming clusters ($\simeq$0.1 Myr old) in a low-metallicity environment ($\simeq$$-$0.7 dex). We performed deep JWST/NIRCam and MIRI imaging of two star-forming clusters in Digel Cloud 2, a star-forming region in the Outer Galaxy ($R_G \gtrsim 15$ kpc). The very high sensitivity and spatial resolution of JWST enable us to resolve cluster members clearly down to a mass detection limit of 0.02 $M_\odot$, enabling the first detection of brown dwarfs in low-metallicity clusters. Fifty-two and ninety-one sources were extracted in mass-$A_V$-limited samples in the two clusters, from which Initial mass functions (IMFs) were derived by model-fitting the F200W band luminosity function, resulting in IMF peak masses (hereafter $M_C$) $\log M_C / M_\odot \simeq -1.5 \pm 0.5$ for both clusters. Although the uncertainties are rather large, the obtained $M_C$ values are lower than those in any previous study ($\log M_C / M_\odot \sim -0.5$). Comparison with the local open clusters with similar ages to the target clusters ($\sim$$10^6$-$10^7$ yr) suggests a metallicity dependence of $M_C$, with lower $M_C$ at lower metallicities, while the comparison with globular clusters, similarly low metallicities but considerably older ($\sim$$10^{10}$ yr), suggests that the target clusters have not yet experienced significant dynamical evolution and remain in their initial physical condition. The lower $M_C$ is also consistent with the theoretical expectation of the lower Jeans mass due to the higher gas density under such low metallicity. The $M_C$ values derived from observations in such an environment would place significant constraints on the understanding of star formation. △ Less

Submitted 27 August, 2024; originally announced August 2024.

Comments: Accepted for publication in ApJ

OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding

Authors: Youjun Zhao, Jiaying Lin, Shuquan Ye, Qianshi Pang, Rynson W. H. Lau

Abstract: Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a mor… ▽ More Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named \textit{OpenScan}, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, and material. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed simply by scaling up object classes during training. We highlight the limitations of existing methodologies and explore promising directions to overcome the identified shortcomings. △ Less

Submitted 9 March, 2025; v1 submitted 20 August, 2024; originally announced August 2024.

Investigating the Electron Capture Supernova Candidate AT 2019abn with JWST Spectroscopy

Authors: Sam Rose, Ryan M. Lau, Jacob E. Jencson, Mansi M. Kasliwal, Kishalay De, Michael E. Ressler, Ori D. Fox, Matthew J. Hankins

Abstract: The James Webb Space Telescope (JWST) has opened up a new window to study highly reddened explosive transients. We present results from late-time (1421 days post-explosion) JWST follow-up spectroscopic observations with NIRSpec and MIRI LRS of the intermediate luminosity red transient (ILRT) AT 2019abn located in the nearby Messier 51 galaxy (8.6 Mpc). ILRTs represent a mysterious class of transie… ▽ More The James Webb Space Telescope (JWST) has opened up a new window to study highly reddened explosive transients. We present results from late-time (1421 days post-explosion) JWST follow-up spectroscopic observations with NIRSpec and MIRI LRS of the intermediate luminosity red transient (ILRT) AT 2019abn located in the nearby Messier 51 galaxy (8.6 Mpc). ILRTs represent a mysterious class of transients which exhibit peak luminosities between those of classical novae and supernovae and which are known to be highly dust obscured. Similar to the prototypical examples of this class of objects, NGC 300 2008-OT and SN 2008S, AT 2019abn has an extremely red and dusty progenitor detected only in pre-explosion Spitzer/IRAC imaging at 3.6 and 4.5 micron and not in deep optical or near-infrared HST images. We find that late time observations of AT 2019abn from NEOWISE and JWST are consistent with the late time evolution of SN 2008S. In part because they are so obscured by dust, it is unknown what produces an ILRT with hypotheses ranging from high mass stellar merger events, non-terminal stellar outbursts, or terminal supernovae explosions through electron-capture in super-AGB stars. Our JWST observations show strong mid-IR Class C PAH features at 6.3 and 8.25 micron typical of carbon-rich post-AGB sources. These features suggest the dust around AT 2019abn, either pre-existing or newly formed in the ejecta, is composed of carbonaceous grains which are not typically observed around red supergiants. However, depending on the strength and temperature of hot bottom burning, SAGBs may be expected to exhibit a carbon-rich chemistry. Thus our JWST observations are consistent with AT 2019abn having an SAGB progenitor. △ Less

Submitted 29 July, 2024; originally announced July 2024.

Comments: 12 pages, 4 figures, submitted to ApJL

An infrared census of R Coronae Borealis Stars II -- Spectroscopic classifications and implications for the rate of low-mass white dwarf mergers

Authors: Viraj R. Karambelkar, Mansi M. Kasliwal, Patrick Tisserand, Shreya Anand, Michael C. B. Ashley, Lars Bildsten, Geoffrey C. Clayton, Courtney C. Crawford, Kishalay De, Nicholas Earley, Matthew J. Hankins, Xander Hall, Astrid Lamberts, Ryan M. Lau, Dan McKenna, Anna Moore, Eran O. Ofek, Roger M. Smith, Roberto Soria, Jamie Soon, Tony Travouillon

Abstract: We present results from a systematic infrared (IR) census of R Coronae Borealis (RCB) stars in the Milky Way, using data from the Palomar Gattini IR (PGIR) survey. R Coronae Borealis stars are dusty, erratic variable stars presumably formed from the merger of a He-core and a CO-core white dwarf (WD). PGIR is a 30 cm $J$-band telescope with a 25 deg$^{2}$ camera that surveys 18000 deg$^{2}$ of the… ▽ More We present results from a systematic infrared (IR) census of R Coronae Borealis (RCB) stars in the Milky Way, using data from the Palomar Gattini IR (PGIR) survey. R Coronae Borealis stars are dusty, erratic variable stars presumably formed from the merger of a He-core and a CO-core white dwarf (WD). PGIR is a 30 cm $J$-band telescope with a 25 deg$^{2}$ camera that surveys 18000 deg$^{2}$ of the northern sky ($δ>-28^{o}$) at a cadence of 2 days. Using PGIR J-band lightcurves for $\sim$60 million stars together with mid-IR colors from WISE, we selected a sample of 530 candidate RCB stars. We obtained near-IR spectra for these candidates and identified 53 RCB stars in our sample. Accounting for our selection criteria, we find that there are a total of $\approx350^{+150}_{-100}$ RCB stars in the Milky Way. Assuming typical RCB lifetimes, this corresponds to an RCB formation rate of 0.8 - 5 $\times$ 10$^{-3}$ yr$^{-1}$, consistent with observational and theoretical estimates of the He-CO WD merger rate. We searched for quasi-periodic pulsations in the PGIR lightcurves of RCB stars and present pulsation periods for 16 RCB stars. We also examined high-cadenced TESS lightcurves for RCB and the chemically similar, but dustless hydrogen-deficient carbon (dLHdC) stars. We find that dLHdC stars show variations on timescales shorter than RCB stars, suggesting that they may have lower masses than RCB stars. Finally, we identified 3 new spectroscopically confirmed and 12 candidate Galactic DY Per type stars - believed to be colder cousins of RCB stars - doubling the sample of Galactic DY Per type stars. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: accepted for publication in PASP

SOFIA/FORCAST Galactic Center Source Catalog

Authors: Angela S. Cotera, Matthew J. Hankins, John Bally, Ashley T. Barnes, Cara D. Battersby, H Perry Hatchfield, Terry L. Herter, Ryan M. Lau, Steven N. Longmore, Elisabeth A. C. Mills, Mark R. Morris, James T. Radomski, Janet P. Simpson, Zachary Stephens, Daniel L. Walker

Abstract: The central regions of the Milky Way constitute a unique laboratory for a wide swath of astrophysical studies, consequently the inner $\sim$400 pc has been the target of numerous large surveys at all accessible wavelengths. In this paper we present a catalog of sources at 25 and 37 $μ$m located within all of the regions observed with the SOFIA/FORCAST instrument in the inner $\sim$200 pc of the Ga… ▽ More The central regions of the Milky Way constitute a unique laboratory for a wide swath of astrophysical studies, consequently the inner $\sim$400 pc has been the target of numerous large surveys at all accessible wavelengths. In this paper we present a catalog of sources at 25 and 37 $μ$m located within all of the regions observed with the SOFIA/FORCAST instrument in the inner $\sim$200 pc of the Galaxy. The majority of the observations were obtained as part of the SOFIA Cycle 7 Galactic Center Legacy program survey, which was designed to complement the Spitzer/MIPS 24 $μ$m catalog in regions saturated in the MIPS observations. Due to the wide variety of source types captured by our observations at 25 and 37 $μ$m, we do not limit the FORCAST source catalog to unresolved point sources, or treat all sources as if they are point-like sources. The catalog includes all detectable sources in the regions, resulting in a catalog of 950 sources, including point sources, compact sources, and extended sources. We also provide the user with metrics to discriminate between the source types. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Comments: 29 pages, 13 figures, Accepted to ApJ

Overview Results of JWST Observations of Star-Forming Clusters in the Extreme Outer Galaxy

Authors: Natsuko Izumi, Michael E. Ressler, Ryan M. Lau, Patrick M. Koch, Masao Saito, Naoto Kobayashi, Chikako Yasui

Abstract: The extreme outer Galaxy (EOG), which we define as the region of the Milky Way with a galactocentric radius of more than 18 kpc, provides an excellent opportunity to study star formation in an environment significantly different from that in the solar neighborhood because of its lower metallicity and lower gas density. We carried out near- and mid-infrared (NIR and MIR) imaging observations toward… ▽ More The extreme outer Galaxy (EOG), which we define as the region of the Milky Way with a galactocentric radius of more than 18 kpc, provides an excellent opportunity to study star formation in an environment significantly different from that in the solar neighborhood because of its lower metallicity and lower gas density. We carried out near- and mid-infrared (NIR and MIR) imaging observations toward two star-forming clusters located in the EOG using JWST NIRCam and MIRI with nine filters: F115W, F150W, F200W, F350W, F405N, F444W, F770W, F1280W, and F2100W. In this paper, we present an overview of the observations, data reduction, and initial results. The NIR sensitivity is approximately 10--80 times better than our previous observation with the Subaru 8.2 m telescope. Accordingly, the mass detection limit reaches to about 0.01--0.05 $M_\odot$, which is about 10 times better than the previous observations. At MIR wavelengths, the high sensitivity and resolution data enable us to resolve individual young stellar objects in such a distant region for the first time. The mass detection limit at MIR F770W filter reaches about 0.1--0.3 $M_\odot$. With these new observations, we have identified components of the clusters that previous surveys did not detect, including class 0 candidates, outflow/jet components, and distinctive nebular structures. These data will enable us to investigate the properties of star formation in the EOG at the same depth of detail as previous observations of star formation in the solar neighborhood. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Comments: 39 pages, 21 figures, and 5 tables. Accepted for publication in Astronomical Journal. The figure with the full resolution is shown in the publication in the Astronomical Journal

Towards quantum-enhanced long-baseline optical/near-IR interferometry

Authors: Jayadev K. Rajagopal, Ryan M. Lau, Isack Padilla, Stephen T. Ridgway, Chaohan Cui, Brittany McClinton, Aqil Sajjad, Stuartt Corder, Mark Rawlings, Fredrik Rantakyro, J. Gabriel Richardson, Amit Ashok, Saikat Guha

Abstract: Microarcsecond resolutions afforded by an optical-NIR array with kilometer-baselines would enable breakthrough science. However significant technology barriers exist in transporting weakly coherent photon states over these distances: primarily photon loss and phase errors. Quantum telescopy, using entangled states to link spatially separated apertures, offers a possible solution to the loss of pho… ▽ More Microarcsecond resolutions afforded by an optical-NIR array with kilometer-baselines would enable breakthrough science. However significant technology barriers exist in transporting weakly coherent photon states over these distances: primarily photon loss and phase errors. Quantum telescopy, using entangled states to link spatially separated apertures, offers a possible solution to the loss of photons. We report on an initiative launched by NSF NOIRLab in collaboration with the Center for Quantum Networks and Arizona Quantum Initiative at the University of Arizona, Tucson, to explore these concepts further. A brief description of the quantum concepts and a possible technology roadmap towards a quantum-enhanced very long baseline optical-NIR interferometric array is presented. An on-sky demonstration of measuring spatial coherence of photons with apertures linked through the simplest Gottesman protocol over short baselines and with limited phase fluctuations is envisaged as the first step. △ Less

Submitted 8 July, 2024; originally announced July 2024.

Comments: Proceeding of SPIE Conference "Astronomical Telescopes + Instrumentation" (June 2024)

Report number: Paper No. 13095-58

Waterfall: Framework for Robust and Scalable Text Watermarking and Provenance for LLMs

Authors: Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, Bryan Kian Hsiang Low

Abstract: Protecting intellectual property (IP) of text such as articles and code is increasingly important, especially as sophisticated attacks become possible, such as paraphrasing by large language models (LLMs) or even unauthorized training of LLMs on copyrighted text to infringe such IP. However, existing text watermarking methods are not robust enough against such attacks nor scalable to millions of u… ▽ More Protecting intellectual property (IP) of text such as articles and code is increasingly important, especially as sophisticated attacks become possible, such as paraphrasing by large language models (LLMs) or even unauthorized training of LLMs on copyrighted text to infringe such IP. However, existing text watermarking methods are not robust enough against such attacks nor scalable to millions of users for practical implementation. In this paper, we propose Waterfall, the first training-free framework for robust and scalable text watermarking applicable across multiple text types (e.g., articles, code) and languages supportable by LLMs, for general text and LLM data provenance. Waterfall comprises several key innovations, such as being the first to use LLM as paraphrasers for watermarking along with a novel combination of techniques that are surprisingly effective in achieving robust verifiability and scalability. We empirically demonstrate that Waterfall achieves significantly better scalability, robust verifiability, and computational efficiency compared to SOTA article-text watermarking methods, and showed how it could be directly applied to the watermarking of code. We also demonstrated that Waterfall can be used for LLM data provenance, where the watermarks of LLM training data can be detected in LLM output, allowing for detection of unauthorized use of data for LLM training and potentially enabling model-centric watermarking of open-sourced LLMs which has been a limitation of existing LLM watermarking works. Our code is available at https://github.com/aoi3142/Waterfall. △ Less

Submitted 29 October, 2024; v1 submitted 5 July, 2024; originally announced July 2024.

Comments: Accepted to EMNLP 2024 Main Conference

Data-Centric AI in the Age of Large Language Models

Authors: Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma, Yao Shu, Jingtan Wang, Xinyuan Niu, Zhenfeng He, Jiangwei Chen, Zijian Zhou, Gregory Kang Ruey Lau, Hieu Dao, Lucas Agussurja, Rachael Hwee Ling Sim, Xiaoqiang Lin, Wenyang Hu, Zhongxiang Dai, Pang Wei Koh, Bryan Kian Hsiang Low

Abstract: This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs). We start by making the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs, and yet it receives disproportionally low attention from the research community. We identify four specific… ▽ More This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs). We start by making the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs, and yet it receives disproportionally low attention from the research community. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization. In each scenario, we underscore the importance of data, highlight promising research directions, and articulate the potential impacts on the research community and, where applicable, the society as a whole. For instance, we advocate for a suite of data-centric benchmarks tailored to the scale and complexity of data for LLMs. These benchmarks can be used to develop new data curation methods and document research efforts and results, which can help promote openness and transparency in AI and LLM research. △ Less

Submitted 20 June, 2024; originally announced June 2024.

Comments: Preprint

MDeRainNet: An Efficient Macro-pixel Image Rain Removal Network

Authors: Tao Yan, Weijiang He, Chenglong Wang, Cihang Wei, Xiangjie Zhu, Yinghui Wang, Rynson W. H. Lau

Abstract: Since rainy weather always degrades image quality and poses significant challenges to most computer vision-based intelligent systems, image de-raining has been a hot research topic. Fortunately, in a rainy light field (LF) image, background obscured by rain streaks in one sub-view may be visible in the other sub-views, and implicit depth information and recorded 4D structural information may benef… ▽ More Since rainy weather always degrades image quality and poses significant challenges to most computer vision-based intelligent systems, image de-raining has been a hot research topic. Fortunately, in a rainy light field (LF) image, background obscured by rain streaks in one sub-view may be visible in the other sub-views, and implicit depth information and recorded 4D structural information may benefit rain streak detection and removal. However, existing LF image rain removal methods either do not fully exploit the global correlations of 4D LF data or only utilize partial sub-views, resulting in sub-optimal rain removal performance and no-equally good quality for all de-rained sub-views. In this paper, we propose an efficient network, called MDeRainNet, for rain streak removal from LF images. The proposed network adopts a multi-scale encoder-decoder architecture, which directly works on Macro-pixel images (MPIs) to improve the rain removal performance. To fully model the global correlation between the spatial and the angular information, we propose an Extended Spatial-Angular Interaction (ESAI) module to merge them, in which a simple and effective Transformer-based Spatial-Angular Interaction Attention (SAIA) block is also proposed for modeling long-range geometric correlations and making full use of the angular information. Furthermore, to improve the generalization performance of our network on real-world rainy scenes, we propose a novel semi-supervised learning framework for our MDeRainNet, which utilizes multi-level KL loss to bridge the domain gap between features of synthetic and real-world rain streaks and introduces colored-residue image guided contrastive regularization to reconstruct rain-free images. Extensive experiments conducted on synthetic and real-world LFIs demonstrate that our method outperforms the state-of-the-art methods both quantitatively and qualitatively. △ Less

Submitted 23 June, 2025; v1 submitted 15 June, 2024; originally announced June 2024.

Comments: 14 pages, 14 figures, 4 tables

The first Palomar Gattini-IR catalog of J-band light curves: construction and public data release

Authors: Shion Murakawa, Kishalay De, Michael C. B. Ashley, Nicholas Earley, Lynne A. Hillenbrand, Mansi M. Kasliwal, Ryan M. Lau, Anna M. Moore, J. L. Sokoloski, Roberto Soria

Abstract: Palomar Gattini-IR (PGIR) is a wide-field, synoptic infrared time domain survey covering $\approx 15000$\,sq.\,deg. of the \textbf{accessible} sky at $\approx 1-3$\,night cadence to a depth of $J\approx 13.0$ and $\approx 14.9$\,Vega mag in and outside the Galactic plane, respectively. Here, we present the first data release of $J$-band light curves of 2MASS sources within the survey footprint cov… ▽ More Palomar Gattini-IR (PGIR) is a wide-field, synoptic infrared time domain survey covering $\approx 15000$\,sq.\,deg. of the \textbf{accessible} sky at $\approx 1-3$\,night cadence to a depth of $J\approx 13.0$ and $\approx 14.9$\,Vega mag in and outside the Galactic plane, respectively. Here, we present the first data release of $J$-band light curves of 2MASS sources within the survey footprint covering approximately the first four years of operations. We describe the construction of the source catalog based on 2MASS point sources, followed by exposure filtering criteria and forced PSF photometry. The catalog contains light curves of $\approx 286$\,million unique sources with 2MASS magnitudes of $J < 15.5$\,mag, with a total of $\approx 50$\,billion photometric measurements and $\approx 20$\,billion individual source detections at signal-to-noise-ratio $> 3$. We demonstrate the photometric fidelity of the catalog by i) quantifying the magnitude-dependent accuracy and uncertainty of the photometry with respect to 2MASS and ii) comparing against forced PGIR aperture photometry for known variable sources. We present simple filtering criteria for selecting reliable photometric measurements as well as example \texttt{Python} notebooks for users. This catalog is textbf{one of} the largest compilation of nightly cadence, synoptic infrared light curves to date, comparable to those in the largest optical surveys, providing a stepping stone to upcoming infrared surveys in the coming decade. △ Less

Submitted 4 April, 2025; v1 submitted 3 June, 2024; originally announced June 2024.

Comments: 10 Pages, 5 figures, submitted to PASP. Full catalog is now available as a tarball at the following link: https://mitprod-my.sharepoint.com/:f:/g/personal/kde1_mit_edu/EkU7NfgTckVMo27cZI2IUcQBGAg2dHADfK8R-8d9RoMhkQ?e=j45BtJ

Journal ref: PASP 136 104501 (2024)

DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors

Authors: Tianyu Huang, Haoze Zhang, Yihan Zeng, Zhilu Zhang, Hui Li, Wangmeng Zuo, Rynson W. H. Lau

Abstract: Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation… ▽ More Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do. △ Less

Submitted 18 December, 2024; v1 submitted 3 June, 2024; originally announced June 2024.

Comments: Accepted by AAAI 2025. Codes are released at: https://github.com/tyhuang0428/DreamPhysics

Color Shift Estimation-and-Correction for Image Enhancement

Authors: Yiyu Li, Ke Xu, Gerhard Petrus Hancke, Rynson W. H. Lau

Abstract: Images captured under sub-optimal illumination conditions may contain both over- and under-exposures. Current approaches mainly focus on adjusting image brightness, which may exacerbate the color tone distortion in under-exposed areas and fail to restore accurate colors in over-exposed regions. We observe that over- and under-exposed regions display opposite color tone distribution shifts with res… ▽ More Images captured under sub-optimal illumination conditions may contain both over- and under-exposures. Current approaches mainly focus on adjusting image brightness, which may exacerbate the color tone distortion in under-exposed areas and fail to restore accurate colors in over-exposed regions. We observe that over- and under-exposed regions display opposite color tone distribution shifts with respect to each other, which may not be easily normalized in joint modeling as they usually do not have ``normal-exposed'' regions/pixels as reference. In this paper, we propose a novel method to enhance images with both over- and under-exposures by learning to estimate and correct such color shifts. Specifically, we first derive the color feature maps of the brightened and darkened versions of the input image via a UNet-based network, followed by a pseudo-normal feature generator to produce pseudo-normal color feature maps. We then propose a novel COlor Shift Estimation (COSE) module to estimate the color shifts between the derived brightened (or darkened) color feature maps and the pseudo-normal color feature maps. The COSE module corrects the estimated color shifts of the over- and under-exposed regions separately. We further propose a novel COlor MOdulation (COMO) module to modulate the separately corrected colors in the over- and under-exposed regions to produce the enhanced image. Comprehensive experiments show that our method outperforms existing approaches. Project webpage: https://github.com/yiyulics/CSEC. △ Less

Submitted 29 May, 2024; v1 submitted 27 May, 2024; originally announced May 2024.

Comments: CVPR2024 accepted paper

WTP19aalnxx: Discovery of a bright mid-infrared transient in the emerging class of low luminosity supernovae revealed by delayed circumstellar interaction

Authors: Charlotte Myers, Kishalay De, Lin Yan, Jacob E. Jencson, Nicholas Earley, Christoffer Fremling, Daichi Hiramatsu, Mansi M. Kasliwal, Ryan M. Lau, Morgan MacLeod, Megan Masterson, Christos Panagiotou, Robert Simcoe, Samaporn Tinyanont

Abstract: While core-collapse supernovae (SNe) often show early and consistent signs of circumstellar (CSM) interaction, some exhibit delayed signatures due to interaction with distant material around the progenitor star. Here we present the discovery in NEOWISE data of WTP19aalnxx, a luminous mid-infrared (IR) transient in the outskirts of the galaxy KUG 0022-007 at $\approx 190$ Mpc. First detected in 201… ▽ More While core-collapse supernovae (SNe) often show early and consistent signs of circumstellar (CSM) interaction, some exhibit delayed signatures due to interaction with distant material around the progenitor star. Here we present the discovery in NEOWISE data of WTP19aalnxx, a luminous mid-infrared (IR) transient in the outskirts of the galaxy KUG 0022-007 at $\approx 190$ Mpc. First detected in 2018, WTP19aalnxx reaches a peak absolute (Vega) magnitude of $\approx-22$ at $4.6 \, μ$m in $\approx3$ yr, comparable to the most luminous interacting SNe. Archival data reveal a $\gtrsim 5\times$ fainter optical counterpart detected since 2015, while follow-up near-IR observations in 2022 reveal an extremely red ($Ks-W2 \approx 3.7$ mag) active transient. Deep optical spectroscopy confirm strong CSM interaction signatures via intermediate-width Balmer emission lines and coronal metal lines. Modeling the broadband spectral energy distribution, we estimate the presence of $\gtrsim 10^{-2}$ M$_\odot$ of warm dust, likely formed in the shock interaction region. Together with the lack of nebular Fe emission, we suggest that WTP19aalnxx is a missed, low (optical) luminosity SN in an emerging family of core-collapse SNe distinguished by their CSM-interaction-powered mid-IR emission that outshines the optical bands. Investigating the Zwicky Transient Facility sample of SNe in NEOWISE data, we find $17$ core-collapse SNe ($\gtrsim 3$% in a volume-limited sample) without early signs of CSM interaction that exhibit delayed IR brightening, suggestive of dense CSM shells at $\lesssim 10^{17}$cm. We suggest that synoptic IR surveys offer a new route to revealing late-time CSM interaction and the prevalence of intense terminal mass loss in massive stars. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: 15 pages, 5 figures, submitted to ApJ

The long-period spectroscopic orbit and dust creation in the Wolf-Rayet binary system WR 125

Authors: Noel D. Richardson, Andrea R. Daly, Peredur M. Williams, Grant M. Hill, Victor I. Shenavrin, Izumi Endo, André-Nicolas Chené, Nicole Karnath, Ryan M. Lau, Anthony F. J. Moffat, Gerd Weigelt

Abstract: Several long-period binaries with a carbon-rich Wolf-Rayet star and an O star produce dust in their wind collisions. In eccentric binaries, this is seen most strongly near periastron passage. The exact conditions leading to dust creation require orbital properties to be determined, which is difficult owing to their long periods. Recently, the binary system WR 125 (WC7+O9III) began a dust creation… ▽ More Several long-period binaries with a carbon-rich Wolf-Rayet star and an O star produce dust in their wind collisions. In eccentric binaries, this is seen most strongly near periastron passage. The exact conditions leading to dust creation require orbital properties to be determined, which is difficult owing to their long periods. Recently, the binary system WR 125 (WC7+O9III) began a dust creation episode seen through an infrared outburst first detected by NEOWISE-R, which was the first outburst detected since 1991. We present new near- and mid-infrared photometry, which we use to show consistency between the two outbursts and derive an orbital period of 28.12$^{+0.10}_{-0.05}$ yr. We use a long time-series of optical spectra to place the first constraints on its orbital elements, on the assumption that this system will produce dust near periastron. The orbit has a mild eccentricity of 0.29$\pm$0.12 and is only derived for the Wolf-Rayet component, as the O star's radial velocities have noise that is likely larger than the expected semi-amplitude of the orbit. We also present SOFIA/FORCAST grism spectroscopy to examine the infrared spectral energy distribution (SED) of the dust during this outburst, comparing its properties to other WCd binaries, deriving a dust temperature of 580 K in 2021. This collection of observations will allow us to plan future observations of this system and place the system in the context of dust-creating Wolf-Rayet binaries. △ Less

Submitted 16 May, 2024; originally announced May 2024.

Comments: accepted to ApJ

PINNACLE: PINN Adaptive ColLocation and Experimental points selection

Authors: Gregory Kang Ruey Lau, Apivich Hemachandra, See-Kiong Ng, Bryan Kian Hsiang Low

Abstract: Physics-Informed Neural Networks (PINNs), which incorporate PDEs as soft constraints, train with a composite loss function that contains multiple training point types: different types of collocation points chosen during training to enforce each PDE and initial/boundary conditions, and experimental points which are usually costly to obtain via experiments or simulations. Training PINNs using this l… ▽ More Physics-Informed Neural Networks (PINNs), which incorporate PDEs as soft constraints, train with a composite loss function that contains multiple training point types: different types of collocation points chosen during training to enforce each PDE and initial/boundary conditions, and experimental points which are usually costly to obtain via experiments or simulations. Training PINNs using this loss function is challenging as it typically requires selecting large numbers of points of different types, each with different training dynamics. Unlike past works that focused on the selection of either collocation or experimental points, this work introduces PINN Adaptive ColLocation and Experimental points selection (PINNACLE), the first algorithm that jointly optimizes the selection of all training point types, while automatically adjusting the proportion of collocation point types as training progresses. PINNACLE uses information on the interaction among training point types, which had not been considered before, based on an analysis of PINN training dynamics via the Neural Tangent Kernel (NTK). We theoretically show that the criterion used by PINNACLE is related to the PINN generalization error, and empirically demonstrate that PINNACLE is able to outperform existing point selection methods for forward, inverse, and transfer learning problems. △ Less

Submitted 11 April, 2024; originally announced April 2024.

Comments: Accepted to 12th International Conference on Learning Representations (ICLR 2024), 36 pages

Temporal-Spatial Processing of Event Camera Data via Delay-Loop Reservoir Neural Network

Authors: Richard Lau, Anthony Tylan-Tyler, Lihan Yao, Rey de Castro Roberto, Robert Taylor, Isaiah Jones

Abstract: This paper describes a temporal-spatial model for video processing with special applications to processing event camera videos. We propose to study a conjecture motivated by our previous study of video processing with delay loop reservoir (DLR) neural network, which we call Temporal-Spatial Conjecture (TSC). The TSC postulates that there is significant information content carried in the temporal r… ▽ More This paper describes a temporal-spatial model for video processing with special applications to processing event camera videos. We propose to study a conjecture motivated by our previous study of video processing with delay loop reservoir (DLR) neural network, which we call Temporal-Spatial Conjecture (TSC). The TSC postulates that there is significant information content carried in the temporal representation of a video signal and that machine learning algorithms would benefit from separate optimization of the spatial and temporal components for intelligent processing. To verify or refute the TSC, we propose a Visual Markov Model (VMM) which decompose the video into spatial and temporal components and estimate the mutual information (MI) of these components. Since computation of video mutual information is complex and time consuming, we use a Mutual Information Neural Network to estimate the bounds of the mutual information. Our result shows that the temporal component carries significant MI compared to that of the spatial component. This finding has often been overlooked in neural network literature. In this paper, we will exploit this new finding to guide our design of a delay-loop reservoir neural network for event camera classification, which results in a 18% improvement on classification accuracy. △ Less

Submitted 12 February, 2024; originally announced March 2024.

Comments: 10 pages, 12 figures, Darpa Distribution Statement A. Approved for public release. Distribution Unlimited

Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields

Authors: Haoyuan Wang, Wenbo Hu, Lei Zhu, Rynson W. H. Lau

Abstract: Inverse rendering aims at recovering both geometry and materials of objects. It provides a more compatible reconstruction for conventional rendering engines, compared with the neural radiance fields (NeRFs). On the other hand, existing NeRF-based inverse rendering methods cannot handle glossy objects with local light interactions well, as they typically oversimplify the illumination as a 2D enviro… ▽ More Inverse rendering aims at recovering both geometry and materials of objects. It provides a more compatible reconstruction for conventional rendering engines, compared with the neural radiance fields (NeRFs). On the other hand, existing NeRF-based inverse rendering methods cannot handle glossy objects with local light interactions well, as they typically oversimplify the illumination as a 2D environmental map, which assumes infinite lights only. Observing the superiority of NeRFs in recovering radiance fields, we propose a novel 5D Neural Plenoptic Function (NeP) based on NeRFs and ray tracing, such that more accurate lighting-object interactions can be formulated via the rendering equation. We also design a material-aware cone sampling strategy to efficiently integrate lights inside the BRDF lobes with the help of pre-filtered radiance fields. Our method has two stages: the geometry of the target object and the pre-filtered environmental radiance fields are reconstructed in the first stage, and materials of the target object are estimated in the second stage with the proposed NeP and material-aware cone sampling strategy. Extensive experiments on the proposed real-world and synthetic datasets demonstrate that our method can reconstruct high-fidelity geometry/materials of challenging glossy objects with complex lighting interactions from nearby objects. Project webpage: https://whyy.site/paper/nep △ Less

Submitted 24 March, 2024; originally announced March 2024.

Comments: CVPR 2024 paper. Project webpage https://whyy.site/paper/nep

ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars

Authors: Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, Rynson W. H. Lau

Abstract: Real-world applications often require a large gallery of 3D assets that share a consistent theme. While remarkable advances have been made in general 3D content creation from text or image, synthesizing customized 3D assets following the shared theme of input 3D exemplars remains an open and challenging problem. In this work, we present ThemeStation, a novel approach for theme-aware 3D-to-3D gener… ▽ More Real-world applications often require a large gallery of 3D assets that share a consistent theme. While remarkable advances have been made in general 3D content creation from text or image, synthesizing customized 3D assets following the shared theme of input 3D exemplars remains an open and challenging problem. In this work, we present ThemeStation, a novel approach for theme-aware 3D-to-3D generation. ThemeStation synthesizes customized 3D assets based on given few exemplars with two goals: 1) unity for generating 3D assets that thematically align with the given exemplars and 2) diversity for generating 3D assets with a high degree of variations. To this end, we design a two-stage framework that draws a concept image first, followed by a reference-informed 3D modeling stage. We propose a novel dual score distillation (DSD) loss to jointly leverage priors from both the input exemplars and the synthesized concept image. Extensive experiments and user studies confirm that ThemeStation surpasses prior works in producing diverse theme-aware 3D models with impressive quality. ThemeStation also enables various applications such as controllable 3D-to-3D generation. △ Less

Submitted 15 May, 2024; v1 submitted 22 March, 2024; originally announced March 2024.

Comments: Accepted to SIGGRAPH 2024. Project page: https://3dthemestation.github.io/

Emission lines due to ionizing radiation from a compact object in the remnant of Supernova 1987A

Authors: C. Fransson, M. J. Barlow, P. J. Kavanagh, J. Larsson, O. C. Jones, B. Sargent, M. Meixner, P. Bouchet, T. Temim, G. S. Wright, J. A. D. L. Blommaert, N. Habel, A. S. Hirschauer, J. Hjorth, L. Lenkić, T. Tikkanen, R. Wesson, A. Coulais, O. D. Fox, R. Gastaud, A. Glasse, J. Jaspers, O. Krause, R. M. Lau, O. Nayak , et al. (9 additional authors not shown)

Abstract: The nearby Supernova 1987A was accompanied by a burst of neutrino emission, which indicates that a compact object (a neutron star or black hole) was formed in the explosion. There has been no direct observation of this compact object. In this work, we observe the supernova remnant with JWST spectroscopy finding narrow infrared emission lines of argon and sulphur. The line emission is spatially unr… ▽ More The nearby Supernova 1987A was accompanied by a burst of neutrino emission, which indicates that a compact object (a neutron star or black hole) was formed in the explosion. There has been no direct observation of this compact object. In this work, we observe the supernova remnant with JWST spectroscopy finding narrow infrared emission lines of argon and sulphur. The line emission is spatially unresolved and blueshifted in velocity relative to the supernova rest frame. We interpret the lines as gas illuminated by a source of ionizing photons located close to the center of the expanding ejecta. Photoionization models show that the line ratios are consistent with ionization by a cooling neutron star or pulsar wind nebula. The velocity shift could be evidence for a neutron star natal kick. △ Less

Submitted 7 March, 2024; originally announced March 2024.

Comments: Authors version of manuscript published in Science on 22 Feb 2024

Journal ref: SCIENCE 22 Feb 2024 Vol 383, Issue 6685 pp. 898-903

Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks

Authors: Yuhao Liu, Zhanghan Ke, Fang Liu, Nanxuan Zhao, Rynson W. H. Lau

Abstract: Diffusion models trained on large-scale datasets have achieved remarkable progress in image synthesis. However, due to the randomness in the diffusion process, they often struggle with handling diverse low-level tasks that require details preservation. To overcome this limitation, we present a new Diff-Plugin framework to enable a single pre-trained diffusion model to generate high-fidelity result… ▽ More Diffusion models trained on large-scale datasets have achieved remarkable progress in image synthesis. However, due to the randomness in the diffusion process, they often struggle with handling diverse low-level tasks that require details preservation. To overcome this limitation, we present a new Diff-Plugin framework to enable a single pre-trained diffusion model to generate high-fidelity results across a variety of low-level tasks. Specifically, we first propose a lightweight Task-Plugin module with a dual branch design to provide task-specific priors, guiding the diffusion process in preserving image content. We then propose a Plugin-Selector that can automatically select different Task-Plugins based on the text instruction, allowing users to edit images by indicating multiple low-level tasks with natural language. We conduct extensive experiments on 8 low-level vision tasks. The results demonstrate the superiority of Diff-Plugin over existing methods, particularly in real-world scenarios. Our ablations further validate that Diff-Plugin is stable, schedulable, and supports robust training across different dataset sizes. △ Less

Submitted 28 May, 2024; v1 submitted 1 March, 2024; originally announced March 2024.

Comments: Accepted to CVPR2024. Replaced some celebrity images to avoid copyright disputes