-
Instruction-Tuned Video-Audio Models Elucidate Functional Specialization in the Brain
Authors:
Subba Reddy Oota,
Khushbu Pahwa,
Prachi Jindal,
Satya Sai Srinath Namburi,
Maneesh Singh,
Tanmoy Chakraborty,
Bapi S. Raju,
Manish Gupta
Abstract:
Recent voxel-wise multimodal brain encoding studies have shown that multimodal large language models (MLLMs) exhibit a higher degree of brain alignment compared to unimodal models in both unimodal and multimodal stimulus settings. More recently, instruction-tuned multimodal models have shown to generate task-specific representations that align strongly with brain activity. However, prior work eval…
▽ More
Recent voxel-wise multimodal brain encoding studies have shown that multimodal large language models (MLLMs) exhibit a higher degree of brain alignment compared to unimodal models in both unimodal and multimodal stimulus settings. More recently, instruction-tuned multimodal models have shown to generate task-specific representations that align strongly with brain activity. However, prior work evaluating the brain alignment of MLLMs has primarily focused on unimodal settings or relied on non-instruction-tuned multimodal models for multimodal stimuli. To address this gap, we investigated brain alignment, that is, measuring the degree of predictivity of neural activity recorded while participants were watching naturalistic movies (video along with audio) with representations derived from MLLMs. We utilized instruction-specific embeddings from six video and two audio instruction-tuned MLLMs. Experiments with 13 video task-specific instructions show that instruction-tuned video MLLMs significantly outperform non-instruction-tuned multimodal (by 15%) and unimodal models (by 20%). Our evaluation of MLLMs for both video and audio tasks using language-guided instructions shows clear disentanglement in task-specific representations from MLLMs, leading to precise differentiation of multimodal functional processing in the brain. We also find that MLLM layers align hierarchically with the brain, with early sensory areas showing strong alignment with early layers, while higher-level visual and language regions align more with middle to late layers. These findings provide clear evidence for the role of task-specific instructions in improving the alignment between brain activity and MLLMs, and open new avenues for mapping joint information processing in both the systems. We make the code publicly available [https://github.com/subbareddy248/mllm_videos].
△ Less
Submitted 9 June, 2025;
originally announced June 2025.
-
Developing cholera outbreak forecasting through qualitative dynamics: Insights into Malawi case study
Authors:
Adrita Ghosh,
Parthasakha Das,
Tanujit Chakraborty,
Pritha Das,
Dibakar Ghosh
Abstract:
Cholera, an acute diarrheal disease, is a serious concern in developing and underdeveloped areas. A qualitative understanding of cholera epidemics aims to foresee transmission patterns based on reported data and mechanistic models. The mechanistic model is a crucial tool for capturing the dynamics of disease transmission and population spread. However, using real-time cholera cases is essential fo…
▽ More
Cholera, an acute diarrheal disease, is a serious concern in developing and underdeveloped areas. A qualitative understanding of cholera epidemics aims to foresee transmission patterns based on reported data and mechanistic models. The mechanistic model is a crucial tool for capturing the dynamics of disease transmission and population spread. However, using real-time cholera cases is essential for forecasting the transmission trend. This prospective study seeks to furnish insights into transmission trends through qualitative dynamics followed by machine learning-based forecasting. The Monte Carlo Markov Chain approach is employed to calibrate the proposed mechanistic model. We identify critical parameters that illustrate the disease's dynamics using partial rank correlation coefficient-based sensitivity analysis. The basic reproduction number as a crucial threshold measures asymptotic dynamics. Furthermore, forward bifurcation directs the stability of the infection state, and Hopf bifurcation suggests that trends in transmission may become unpredictable as societal disinfection rates rise. Further, we develop epidemic-informed machine learning models by incorporating mechanistic cholera dynamics into autoregressive integrated moving averages and autoregressive neural networks. We forecast short-term future cholera cases in Malawi by implementing the proposed epidemic-informed machine learning models to support this. We assert that integrating temporal dynamics into the machine learning models can enhance the capabilities of cholera forecasting models. The execution of this mechanism can significantly influence future trends in cholera transmission. This evolving approach can also be beneficial for policymakers to interpret and respond to potential disease systems. Moreover, our methodology is replicable and adaptable, encouraging future research on disease dynamics.
△ Less
Submitted 18 March, 2025;
originally announced March 2025.
-
Epidemic-guided deep learning for spatiotemporal forecasting of Tuberculosis outbreak
Authors:
Madhab Barman,
Madhurima Panja,
Nachiketa Mishra,
Tanujit Chakraborty
Abstract:
Tuberculosis (TB) remains a formidable global health challenge, driven by complex spatiotemporal transmission dynamics and influenced by factors such as population mobility and behavioral changes. We propose an Epidemic-Guided Deep Learning (EGDL) approach that fuses mechanistic epidemiological principles with advanced deep learning techniques to enhance early warning systems and intervention stra…
▽ More
Tuberculosis (TB) remains a formidable global health challenge, driven by complex spatiotemporal transmission dynamics and influenced by factors such as population mobility and behavioral changes. We propose an Epidemic-Guided Deep Learning (EGDL) approach that fuses mechanistic epidemiological principles with advanced deep learning techniques to enhance early warning systems and intervention strategies for TB outbreaks. Our framework is built upon a modified networked Susceptible-Infectious-Recovered (MN-SIR) model augmented with a saturated incidence rate and graph Laplacian diffusion, capturing both long-term transmission dynamics and region-specific population mobility patterns. Compartmental model parameters are rigorously estimated using Bayesian inference via the Markov Chain Monte Carlo approach. Theoretical analysis leveraging the comparison principle and Green's formula establishes global stability properties of the disease-free and endemic equilibria. Building on these epidemiological insights, we design two forecasting architectures, EGDL-Parallel and EGDL-Series, that integrate the mechanistic outputs of the MN-SIR model within deep neural networks. This integration mitigates the overfitting risks commonly encountered in data-driven methods and filters out noise inherent in surveillance data, resulting in reliable forecasts of real-world epidemic trends. Experiments conducted on TB incidence data from 47 prefectures in Japan and 31 provinces in mainland China demonstrate that our approach delivers robust and accurate predictions across multiple time horizons (short to medium-term forecasts), supporting its generalizability across regions with different population dynamics.
△ Less
Submitted 8 June, 2025; v1 submitted 15 February, 2025;
originally announced February 2025.
-
An ensemble neural network approach to forecast Dengue outbreak based on climatic condition
Authors:
Madhurima Panja,
Tanujit Chakraborty,
Sk Shahid Nadim,
Indrajit Ghosh,
Uttam Kumar,
Nan Liu
Abstract:
Dengue fever is a virulent disease spreading over 100 tropical and subtropical countries in Africa, the Americas, and Asia. This arboviral disease affects around 400 million people globally, severely distressing the healthcare systems. The unavailability of a specific drug and ready-to-use vaccine makes the situation worse. Hence, policymakers must rely on early warning systems to control interven…
▽ More
Dengue fever is a virulent disease spreading over 100 tropical and subtropical countries in Africa, the Americas, and Asia. This arboviral disease affects around 400 million people globally, severely distressing the healthcare systems. The unavailability of a specific drug and ready-to-use vaccine makes the situation worse. Hence, policymakers must rely on early warning systems to control intervention-related decisions. Forecasts routinely provide critical information for dangerous epidemic events. However, the available forecasting models (e.g., weather-driven mechanistic, statistical time series, and machine learning models) lack a clear understanding of different components to improve prediction accuracy and often provide unstable and unreliable forecasts. This study proposes an ensemble wavelet neural network with exogenous factor(s) (XEWNet) model that can produce reliable estimates for dengue outbreak prediction for three geographical regions, namely San Juan, Iquitos, and Ahmedabad. The proposed XEWNet model is flexible and can easily incorporate exogenous climate variable(s) confirmed by statistical causality tests in its scalable framework. The proposed model is an integrated approach that uses wavelet transformation into an ensemble neural network framework that helps in generating more reliable long-term forecasts. The proposed XEWNet allows complex non-linear relationships between the dengue incidence cases and rainfall; however, mathematically interpretable, fast in execution, and easily comprehensible. The proposal's competitiveness is measured using computational experiments based on various statistical metrics and several statistical comparison tests. In comparison with statistical, machine learning, and deep learning methods, our proposed XEWNet performs better in 75% of the cases for short-term and long-term forecasting of dengue incidence.
△ Less
Submitted 19 December, 2022; v1 submitted 16 December, 2022;
originally announced December 2022.
-
Epicasting: An Ensemble Wavelet Neural Network (EWNet) for Forecasting Epidemics
Authors:
Madhurima Panja,
Tanujit Chakraborty,
Uttam Kumar,
Nan Liu
Abstract:
Infectious diseases remain among the top contributors to human illness and death worldwide, among which many diseases produce epidemic waves of infection. The unavailability of specific drugs and ready-to-use vaccines to prevent most of these epidemics makes the situation worse. These force public health officials and policymakers to rely on early warning systems generated by reliable and accurate…
▽ More
Infectious diseases remain among the top contributors to human illness and death worldwide, among which many diseases produce epidemic waves of infection. The unavailability of specific drugs and ready-to-use vaccines to prevent most of these epidemics makes the situation worse. These force public health officials and policymakers to rely on early warning systems generated by reliable and accurate forecasts of epidemics. Accurate forecasts of epidemics can assist stakeholders in tailoring countermeasures, such as vaccination campaigns, staff scheduling, and resource allocation, to the situation at hand, which could translate to reductions in the impact of a disease. Unfortunately, most of these past epidemics exhibit nonlinear and non-stationary characteristics due to their spreading fluctuations based on seasonal-dependent variability and the nature of these epidemics. We analyse a wide variety of epidemic time series datasets using a maximal overlap discrete wavelet transform (MODWT) based autoregressive neural network and call it EWNet model. MODWT techniques effectively characterize non-stationary behavior and seasonal dependencies in the epidemic time series and improve the nonlinear forecasting scheme of the autoregressive neural network in the proposed ensemble wavelet network framework. From a nonlinear time series viewpoint, we explore the asymptotic stationarity of the proposed EWNet model to show the asymptotic behavior of the associated Markov Chain. We also theoretically investigate the effect of learning stability and the choice of hidden neurons in the proposal. From a practical perspective, we compare our proposed EWNet framework with several statistical, machine learning, and deep learning models. Experimental results show that the proposed EWNet is highly competitive compared to the state-of-the-art epidemic forecasting methods.
△ Less
Submitted 14 March, 2023; v1 submitted 21 June, 2022;
originally announced June 2022.
-
Nowcasting of COVID-19 confirmed cases: Foundations, trends, and challenges
Authors:
Tanujit Chakraborty,
Indrajit Ghosh,
Tirna Mahajan,
Tejasvi Arora
Abstract:
The coronavirus disease 2019 (COVID-19) has become a public health emergency of international concern affecting more than 200 countries and territories worldwide. As of September 30, 2020, it has caused a pandemic outbreak with more than 33 million confirmed infections and more than 1 million reported deaths worldwide. Several statistical, machine learning, and hybrid models have previously tried…
▽ More
The coronavirus disease 2019 (COVID-19) has become a public health emergency of international concern affecting more than 200 countries and territories worldwide. As of September 30, 2020, it has caused a pandemic outbreak with more than 33 million confirmed infections and more than 1 million reported deaths worldwide. Several statistical, machine learning, and hybrid models have previously tried to forecast COVID-19 confirmed cases for profoundly affected countries. Due to extreme uncertainty and nonstationarity in the time series data, forecasting of COVID-19 confirmed cases has become a very challenging job. For univariate time series forecasting, there are various statistical and machine learning models available in the literature. But, epidemic forecasting has a dubious track record. Its failures became more prominent due to insufficient data input, flaws in modeling assumptions, high sensitivity of estimates, lack of incorporation of epidemiological features, inadequate past evidence on effects of available interventions, lack of transparency, errors, lack of determinacy, and lack of expertise in crucial disciplines. This chapter focuses on assessing different short-term forecasting models that can forecast the daily COVID-19 cases for various countries. In the form of an empirical study on forecasting accuracy, this chapter provides evidence to show that there is no universal method available that can accurately forecast pandemic data. Still, forecasters' predictions are useful for the effective allocation of healthcare resources and will act as an early-warning system for government policymakers.
△ Less
Submitted 10 October, 2020;
originally announced October 2020.
-
How Reliable is Smartphone-based Electronic Contact Tracing for COVID-19?
Authors:
Philipp H. Kindt,
Trinad Chakraborty,
Samarjit Chakraborty
Abstract:
Smartphone-based electronic contact tracing is currently considered an essential tool towards easing lockdowns, curfews, and shelter-in-place orders issued by most governments around the world in response to the 2020 novel coronavirus (SARS-CoV-2) crisis. While the focus on developing smartphone-based contact tracing applications or apps has been on privacy concerns stemming from the use of such a…
▽ More
Smartphone-based electronic contact tracing is currently considered an essential tool towards easing lockdowns, curfews, and shelter-in-place orders issued by most governments around the world in response to the 2020 novel coronavirus (SARS-CoV-2) crisis. While the focus on developing smartphone-based contact tracing applications or apps has been on privacy concerns stemming from the use of such apps, an important question that has not received sufficient attention is: How reliable will such smartphone-based electronic contact tracing be?
This is a technical question related to how two smartphones reliably register their mutual proximity. Here, we examine in detail the technical prerequisites required for effective smartphone-based contact tracing. The underlying mechanism that any contact tracing app relies on is called Neighbor Discovery (ND), which involves smartphones transmitting and scanning for Bluetooth signals to record their mutual presence whenever they are in close proximity. The hardware support and the software protocols used for ND in smartphones, however, were not designed for reliable contact tracing. In this paper, we quantitatively evaluate how reliably can smartphones do contact tracing. Our results point towards the design of a wearable solution for contact tracing that can overcome the shortcomings of a smartphone-based solution to provide more reliable and accurate contact tracing. To the best of our knowledge, this is the first study that quantifies, both, the suitability and also the drawbacks of smartphone-based contact tracing. Further, our results can be used to parameterize a ND protocol to maximize the reliability of any contact tracing app that uses it.
△ Less
Submitted 22 May, 2020; v1 submitted 12 May, 2020;
originally announced May 2020.
-
Coronavirus (COVID-19): ARIMA based time-series analysis to forecast near future
Authors:
Hiteshi Tandon,
Prabhat Ranjan,
Tanmoy Chakraborty,
Vandana Suhag
Abstract:
COVID-19, a novel coronavirus, is currently a major worldwide threat. It has infected more than a million people globally leading to hundred-thousands of deaths. In such grave circumstances, it is very important to predict the future infected cases to support prevention of the disease and aid in the healthcare service preparation. Following that notion, we have developed a model and then employed…
▽ More
COVID-19, a novel coronavirus, is currently a major worldwide threat. It has infected more than a million people globally leading to hundred-thousands of deaths. In such grave circumstances, it is very important to predict the future infected cases to support prevention of the disease and aid in the healthcare service preparation. Following that notion, we have developed a model and then employed it for forecasting future COVID-19 cases in India. The study indicates an ascending trend for the cases in the coming days. A time series analysis also presents an exponential increase in the number of cases. It is supposed that the present prediction models will assist the government and medical personnel to be prepared for the upcoming conditions and have more readiness in healthcare systems.
△ Less
Submitted 16 April, 2020;
originally announced April 2020.
