-
Scientific Hypothesis Generation and Validation: Methods, Datasets, and Future Directions
Authors:
Adithya Kulkarni,
Fatimah Alotaibi,
Xinyue Zeng,
Longfeng Wu,
Tong Zeng,
Barry Menglong Yao,
Minqian Liu,
Shuaicheng Zhang,
Lifu Huang,
Dawei Zhou
Abstract:
Large Language Models (LLMs) are transforming scientific hypothesis generation and validation by enabling information synthesis, latent relationship discovery, and reasoning augmentation. This survey provides a structured overview of LLM-driven approaches, including symbolic frameworks, generative models, hybrid systems, and multi-agent architectures. We examine techniques such as retrieval-augmen…
▽ More
Large Language Models (LLMs) are transforming scientific hypothesis generation and validation by enabling information synthesis, latent relationship discovery, and reasoning augmentation. This survey provides a structured overview of LLM-driven approaches, including symbolic frameworks, generative models, hybrid systems, and multi-agent architectures. We examine techniques such as retrieval-augmented generation, knowledge-graph completion, simulation, causal inference, and tool-assisted reasoning, highlighting trade-offs in interpretability, novelty, and domain alignment. We contrast early symbolic discovery systems (e.g., BACON, KEKADA) with modern LLM pipelines that leverage in-context learning and domain adaptation via fine-tuning, retrieval, and symbolic grounding. For validation, we review simulation, human-AI collaboration, causal modeling, and uncertainty quantification, emphasizing iterative assessment in open-world contexts. The survey maps datasets across biomedicine, materials science, environmental science, and social science, introducing new resources like AHTech and CSKG-600. Finally, we outline a roadmap emphasizing novelty-aware generation, multimodal-symbolic integration, human-in-the-loop systems, and ethical safeguards, positioning LLMs as agents for principled, scalable scientific discovery.
△ Less
Submitted 6 May, 2025;
originally announced May 2025.
-
A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
Authors:
Zihao Lin,
Samyadeep Basu,
Mohammad Beigi,
Varun Manjunatha,
Ryan A. Rossi,
Zichao Wang,
Yufan Zhou,
Sriram Balasubramanian,
Arman Zarei,
Keivan Rezaei,
Ying Shen,
Barry Menglong Yao,
Zhiyang Xu,
Qin Liu,
Yuxiang Zhang,
Yan Sun,
Shilong Liu,
Li Shen,
Hongxuan Li,
Soheil Feizi,
Lifu Huang
Abstract:
The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting Large Language Models (LLMs), multimodal foundation models (MMFMs) - such as contrastive vision-language models, generative vision-language models,…
▽ More
The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting Large Language Models (LLMs), multimodal foundation models (MMFMs) - such as contrastive vision-language models, generative vision-language models, and text-to-image models - pose unique interpretability challenges beyond unimodal frameworks. Despite initial studies, a substantial gap remains between the interpretability of LLMs and MMFMs. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal language models and crossmodal systems. By systematically reviewing current MMFM analysis techniques, we propose a structured taxonomy of interpretability methods, compare insights across unimodal and multimodal architectures, and highlight critical research gaps.
△ Less
Submitted 22 February, 2025;
originally announced February 2025.
-
Error-driven Data-efficient Large Multimodal Model Tuning
Authors:
Barry Menglong Yao,
Qifan Wang,
Lifu Huang
Abstract:
Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framewo…
▽ More
Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without requiring any task-specific training samples. In our approach, a generic LMM, acting as a student model, is first evaluated on a small validation set of the target task, and then a more powerful model, acting as a teacher model, identifies the erroneous steps within the student model's reasoning steps and analyzes its capability gaps from fully addressing the target task. Based on these gaps, targeted training samples are further retrieved from existing task-agnostic datasets to tune the student model and tailor it to the target task. We perform extensive experiments across three different training data scales and seven tasks, demonstrating that our training paradigm significantly and efficiently improves LMM's performance on downstream tasks, achieving an average performance boost of 7.01%.
△ Less
Submitted 20 December, 2024;
originally announced December 2024.
-
AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes
Authors:
Barry Menglong Yao,
Sijia Wang,
Yu Chen,
Qifan Wang,
Minqian Liu,
Zhiyang Xu,
Licheng Yu,
Lifu Huang
Abstract:
We propose attribute-aware multimodal entity linking, where the input consists of a mention described with a text paragraph and images, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also accompanied by a text description, visual images, and a collection of attributes that present the meta-information of the entity in a structu…
▽ More
We propose attribute-aware multimodal entity linking, where the input consists of a mention described with a text paragraph and images, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also accompanied by a text description, visual images, and a collection of attributes that present the meta-information of the entity in a structured format. To facilitate this research endeavor, we construct AMELI, encompassing a new multimodal entity linking benchmark dataset that contains 16,735 mentions described in text and associated with 30,472 images, and a multimodal knowledge base that covers 34,690 entities along with 177,873 entity images and 798,216 attributes. To establish baseline performance on AMELI, we experiment with several state-of-the-art architectures for multimodal entity linking and further propose a new approach that incorporates attributes of entities into disambiguation. Experimental results and extensive qualitative analysis demonstrate that extracting and understanding the attributes of mentions from their text descriptions and visual images play a vital role in multimodal entity linking. To the best of our knowledge, we are the first to integrate attributes in the multimodal entity linking task. The programs, model checkpoints, and the dataset are publicly available at https://github.com/VT-NLP/Ameli.
△ Less
Submitted 10 June, 2025; v1 submitted 24 May, 2023;
originally announced May 2023.
-
End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models
Authors:
Barry Menglong Yao,
Aditya Shah,
Lichao Sun,
Jin-Hee Cho,
Lifu Huang
Abstract:
We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summa…
▽ More
We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e.g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims where each claim is annotated with a truthfulness label and a ruling statement, and 33,880 textual paragraphs and 12,112 images in total as evidence. To establish baseline performances on Mocheg, we experiment with several state-of-the-art neural architectures on the three pipelined subtasks: multimodal evidence retrieval, claim verification, and explanation generation, and demonstrate that the performance of the state-of-the-art end-to-end multimodal fact-checking does not provide satisfactory outcomes. To the best of our knowledge, we are the first to build the benchmark dataset and solutions for end-to-end multimodal fact-checking and explanation generation. The dataset, source code and model checkpoints are available at https://github.com/VT-NLP/Mocheg.
△ Less
Submitted 6 July, 2023; v1 submitted 25 May, 2022;
originally announced May 2022.
-
Unconventional Singularity in Anti-Parity-Time Symmetric Cavity Magnonics
Authors:
Y. Yang,
Yi-Pu Wang,
J. W. Rao,
Y. S. Gui,
B. M. Yao,
W. Lu,
C. -M. Hu
Abstract:
By engineering an anti-parity-time (anti-PT) symmetric cavity magnonics system with precise eigenspace controllability, we observe two different singularities in the same system. One type of singularity, the exceptional point (EP), is produced by tuning the magnon damping. Between two EPs, the maximal coherent superposition of photon and magnon states is robustly sustained by the preserved anti-PT…
▽ More
By engineering an anti-parity-time (anti-PT) symmetric cavity magnonics system with precise eigenspace controllability, we observe two different singularities in the same system. One type of singularity, the exceptional point (EP), is produced by tuning the magnon damping. Between two EPs, the maximal coherent superposition of photon and magnon states is robustly sustained by the preserved anti-PT symmetry. The other type of singularity, arising from the dissipative coupling of two anti-resonances, is an unconventional bound state in the continuum (BIC). At the settings of BICs, the coupled system exhibits infinite discontinuities in the group delay. We find that both singularities co-exist at the equator of the Bloch sphere, which reveals a unique hybrid state that simultaneously exhibits the maximal coherent superposition and slow light capability.
△ Less
Submitted 8 September, 2020;
originally announced September 2020.
-
Nonreciprocity and Unidirectional Invisibility in Cavity Magnonics
Authors:
Yi-Pu Wang,
J. W. Rao,
Y. Yang,
Peng-Chao Xu,
Y. S. Gui,
B. M. Yao,
J. Q. You,
C. -M. Hu
Abstract:
We reveal the cooperative effect of coherent and dissipative magnon-photon couplings in an open cavity magnonic system, which leads to nonreciprocity with a considerably large isolation ratio and flexible controllability. Furthermore, we discover unidirectional invisibility for microwave propagation, which appears at the zero-damping condition for hybrid magnon-photon modes. A simple model is deve…
▽ More
We reveal the cooperative effect of coherent and dissipative magnon-photon couplings in an open cavity magnonic system, which leads to nonreciprocity with a considerably large isolation ratio and flexible controllability. Furthermore, we discover unidirectional invisibility for microwave propagation, which appears at the zero-damping condition for hybrid magnon-photon modes. A simple model is developed to capture the generic physics of the interference between coherent and dissipative couplings, which accurately reproduces the observations over a broad range of parameters. This general scheme could inspire methods to achieve nonreciprocity in other systems.
△ Less
Submitted 21 August, 2019;
originally announced August 2019.
-
Coherent control of magnon radiative damping with local photon states
Authors:
B. M. Yao,
T. Yu,
Y. S. Gui,
J. W. Rao,
Y. T. Zhao,
W. Lu,
C. -M. Hu
Abstract:
The collective excitation of ordered spins, known as spin waves or magnons, can in principle radiate by emitting travelling photons to an open system when decaying to the ground state. However, in contrast to the electric dipoles, magnetic dipoles contributed by magnons are more isolated from electromagnetic environment with negligible radiation in the vacuum, limiting their application in coheren…
▽ More
The collective excitation of ordered spins, known as spin waves or magnons, can in principle radiate by emitting travelling photons to an open system when decaying to the ground state. However, in contrast to the electric dipoles, magnetic dipoles contributed by magnons are more isolated from electromagnetic environment with negligible radiation in the vacuum, limiting their application in coherent communication by photons. Recently, strong interaction between cavity standing-wave photons and magnons has been reported, indicating the possible manipulation of magnon radiation via tailoring photon states. Here, with loading an yttrium iron garnet sphere in a one-dimensional circular waveguide cavity in the presence of both travelling and standing photon modes, we demonstrate an efficient photon emissions from magnon and a significant magnon radiative damping with radiation rate found to be proportional to the local density of states (LDOS) of photon. By modulating the LDOS including its magnitude and/or polarization, we can flexibly tune the photon emission and magnon radiative damping on demand. Our findings provide a general way in manipulating photon emission from magnon radiation for harnessing energy and angular momentum generation, transfer and storage modulated by magnon in the cavity and waveguide electrodynamics.
△ Less
Submitted 10 September, 2019; v1 submitted 18 February, 2019;
originally announced February 2019.
-
Control of the magnon-photon level attraction in a planar cavity
Authors:
Y. Yang,
J. W. Rao,
Y. S. Gui,
B. M. Yao,
W. Lu,
C. -M. Hu
Abstract:
A resistive coupling circuit is used to model the recently discovered dissipative coupling in a hybridized cavity photon-magnon system. With this model as a basis we have designed a planar cavity in which a controllable transition between level attraction and level repulsion can be achieved. This behaviour can be quantitatively understood using an LCR circuit model with a complex coupling strength…
▽ More
A resistive coupling circuit is used to model the recently discovered dissipative coupling in a hybridized cavity photon-magnon system. With this model as a basis we have designed a planar cavity in which a controllable transition between level attraction and level repulsion can be achieved. This behaviour can be quantitatively understood using an LCR circuit model with a complex coupling strength. Our work therefore develops and verifies a circuit method to model level repulsion and level attraction and confirms the universality of dissipative coupling in the cavity photon-magnon system. The realization of both coherent and dissipative couplings in a planar cavity may provide new avenues for the design and adaptation of dissipatively coupled systems for practical applications in information processing.
△ Less
Submitted 8 April, 2019; v1 submitted 22 January, 2019;
originally announced January 2019.
-
Level Attraction Due to Dissipative Magnon-Photon Coupling
Authors:
M. Harder,
Y. Yang,
B. M. Yao,
C. H. Yu,
J. W. Rao,
Y. S. Gui,
R. L. Stamps,
C. -M. Hu
Abstract:
We report dissipative magnon-photon coupling caused by cavity Lenz effect, where the magnons in a magnet induce a rf current in the cavity, leading to a cavity back action that impedes the magnetization dynamics. This effect is revealed in our experiment as level attraction with a coalescence of hybridized magnon-photon modes, which is distinctly different from level repulsion with mode anticrossi…
▽ More
We report dissipative magnon-photon coupling caused by cavity Lenz effect, where the magnons in a magnet induce a rf current in the cavity, leading to a cavity back action that impedes the magnetization dynamics. This effect is revealed in our experiment as level attraction with a coalescence of hybridized magnon-photon modes, which is distinctly different from level repulsion with mode anticrossing caused by coherent magnon-photon coupling. We develop a method to control the in- terpolation of coherent and dissipative magnon-photon coupling, and observe a matching condition where the two effects cancel. Our work sheds light on the so-far hidden side of magnon-photon coupling, opening a new avenue for controlling and utilizing light-matter interactions.
△ Less
Submitted 4 September, 2018;
originally announced September 2018.
-
Theory and experiment on cavity magnon polariton in the 1D configuration
Authors:
B. M. Yao,
Y. S. Gui,
Y. Xiao,
H. Guo,
X. S. Chen,
W. Lu,
C. L. Chien,
C. -M. Hu
Abstract:
We have theoretically and experimentally investigated the dispersion of the cavity-magnon-polariton (CMP) in a 1D configuration, created by inserting a low damping magnetic insulator into a high-quality 1D microwave cavity. By simplifying the full-wave simulation based on the transfer matrix approach in the long wavelength limit, an analytic approximation of the CMP dispersion has been obtained. T…
▽ More
We have theoretically and experimentally investigated the dispersion of the cavity-magnon-polariton (CMP) in a 1D configuration, created by inserting a low damping magnetic insulator into a high-quality 1D microwave cavity. By simplifying the full-wave simulation based on the transfer matrix approach in the long wavelength limit, an analytic approximation of the CMP dispersion has been obtained. The resultant coupling strength of the CMP shows different dependence on the sample thickness as well as the permittivity of the sample, determined by the parity of the cavity modes. These scaling effects of the cavity and material parameters are confirmed by experimental data. Our work provide a detailed understanding of the 1D CMP, which could help to engineer coupled magnon-photon system.
△ Less
Submitted 18 September, 2015;
originally announced September 2015.
-
On-chip artificial magnon-polariton device for voltage control of electromagnetically induced transparency
Authors:
S. Kaur,
B. M. Yao,
Y. S. Gui,
C. -M. Hu
Abstract:
We demonstrate an on-chip device utilizing the concept of artificial cavity magnon-polariton (CMP) coupling between the microwave cavity mode and the dynamics of the artificial magnetism in a split ring resonator. This on-chip device allows the easy tuning of the artificial CMP gap by using a DC voltage signal, which enables tuneable electrodynamically induced transparency. The high tunability of…
▽ More
We demonstrate an on-chip device utilizing the concept of artificial cavity magnon-polariton (CMP) coupling between the microwave cavity mode and the dynamics of the artificial magnetism in a split ring resonator. This on-chip device allows the easy tuning of the artificial CMP gap by using a DC voltage signal, which enables tuneable electrodynamically induced transparency. The high tunability of the artificial magnon-polariton system not only enables the study of the characteristic phenomena associated with distinct coupling regimes, but also may open up avenues for designing novel microwave devices and ultra-sensitive sensors.
△ Less
Submitted 24 July, 2015;
originally announced July 2015.
-
Quantifying the complex permittivity and permeability of magnetic nanoparticles
Authors:
B. M. Yao,
Y. S. Gui,
M. Worden,
T. Hegmann,
M. Xing,
X. S. Chen,
W. Lu,
Y. Wroczynskyj,
J. van Lierop,
C. -M. Hu
Abstract:
The complex permittivity and permeability of superparamagnetic iron-oxide nanoparticles has been quantified using a circular waveguide assembly with a static magnetic field to align the nanoparticle's magnetization. The high sensitivity of the measurement provides the precise resonant feature of nanoparticles. The complex permeability in the vicinity of ferromagnetic resonance (FMR) is in agreemen…
▽ More
The complex permittivity and permeability of superparamagnetic iron-oxide nanoparticles has been quantified using a circular waveguide assembly with a static magnetic field to align the nanoparticle's magnetization. The high sensitivity of the measurement provides the precise resonant feature of nanoparticles. The complex permeability in the vicinity of ferromagnetic resonance (FMR) is in agreement with the nanoparticle's measured magnetization via conventional magnetometry. A rigorous and self-consistent measure of complex permittivities and permeabilities of nanoparticles is crucial to ascertain accurately the dielectric behaviour as well as the frequency response of nanoparticle magnetization, necessary ingredients when designing and optimizing magnetic nanoparticles for biomedical applications.
△ Less
Submitted 6 April, 2015;
originally announced April 2015.
-
Rapid microwave phase detection based on a solid state spiontronic device
Authors:
B. M. Yao,
L. Fu,
X. S. Chen,
W. Lu,
L. H. Bai,
Y. S. Gui,
C. -M. Hu
Abstract:
A technique for rapidly detecting microwave phase has been developed which uses a spintronic device that can directly rectify microwave fields into a dc voltage signal. Use of a voltage-controlled phase shifter enables the development of a spintronic device that can simultaneously 'read' the magnitude and phase of incident continuous-wave (CW) microwaves when combined with a lock-in amplifier. As…
▽ More
A technique for rapidly detecting microwave phase has been developed which uses a spintronic device that can directly rectify microwave fields into a dc voltage signal. Use of a voltage-controlled phase shifter enables the development of a spintronic device that can simultaneously 'read' the magnitude and phase of incident continuous-wave (CW) microwaves when combined with a lock-in amplifier. As an example of many possible practical applications of this device, the resonance phase in a complementary electric inductive-capacitive (CELC) resonator has been characterized using a spintronic sensor based on a magnetic tunnel junction (MTJ). This sensor device is not limited for use only with spintronic devices such as MTJs, but can also be used with semiconductor devices such as microwave detectors, and hence offers a useful alternative to existing microwave imaging and characterization technologies.
△ Less
Submitted 13 September, 2013;
originally announced September 2013.
