-
A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection
Authors:
Morteza Bodaghi,
Majid Hosseini,
Raju Gottumukkala
Abstract:
Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learnin…
▽ More
Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learning by simplifying data and making the features more amenable to processing and analysis, thereby reducing computational complexity. This paper introduces an intermediate multimodal fusion network with manifold learning-based dimensionality reduction. The multimodal network generates independent representations from biometric signals and facial landmarks through 1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN layer, followed by a fully connected dense layer. We compared various dimensionality reduction techniques for different variations of unimodal and multimodal networks. We observe that the intermediate-level fusion with the Multi-Dimensional Scaling (MDS) manifold method showed promising results with an accuracy of 96.00\% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV) paradigm over other dimensional reduction methods. MDS had the highest computational cost among manifold learning methods. However, while outperforming other networks, it managed to reduce the computational cost of the proposed networks by 25\% when compared to six well-known conventional feature selection methods used in the preprocessing step.
△ Less
Submitted 12 March, 2024;
originally announced March 2024.
-
Multimodal Stress Detection Using Facial Landmarks and Biometric Signals
Authors:
Majid Hosseini,
Morteza Bodaghi,
Ravi Teja Bhupatiraju,
Anthony Maida,
Raju Gottumukkala
Abstract:
The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning…
▽ More
The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39\% accuracy, and early-fusion surpassed it with a 98.38\% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
Analysis of the Effectiveness of Face-Coverings on the Death Ratio of COVID-19 Using Machine Learning
Authors:
Ali Lafzi,
Miad Boodaghi,
Siavash Zamani,
Niyousha Mohammadshafie,
Veeraraghava Raju Hasti
Abstract:
The recent outbreak of the COVID-19 led to the death of millions of people worldwide. To stave off the spread of the virus, the authorities in the US employed different strategies, including the mask mandate order issued by the states' governors. In the current work, we defined a parameter called the average death ratio as the monthly average of the number of daily deaths to the monthly average nu…
▽ More
The recent outbreak of the COVID-19 led to the death of millions of people worldwide. To stave off the spread of the virus, the authorities in the US employed different strategies, including the mask mandate order issued by the states' governors. In the current work, we defined a parameter called the average death ratio as the monthly average of the number of daily deaths to the monthly average number of daily cases. We utilized survey data to quantify people's abidance by the mask mandate order. Additionally, we implicitly addressed the extent to which people abide by the mask mandate order that may depend on some parameters like population, income, and education level. Using different machine learning classification algorithms, we investigated how the decrease or increase in death ratio for the counties in the US West Coast correlates with the input parameters. The results showed that for most counties there, the mask mandate order decreased the death ratio reflecting the effectiveness of this preventive measure on the West Coast. Additionally, the changes in the death ratio demonstrated a noticeable correlation with the socio-economic condition of each county. Moreover, the results showed a promising classification accuracy score as high as around 90%.
△ Less
Submitted 7 November, 2021; v1 submitted 8 February, 2021;
originally announced February 2021.
