-
Hamiltonian Adaptive Importance Sampling
Authors:
Ali Mousavi,
Reza Monsefi,
Víctor Elvira
Abstract:
Importance sampling (IS) is a powerful Monte Carlo (MC) methodology for approximating integrals, for instance in the context of Bayesian inference. In IS, the samples are simulated from the so-called proposal distribution, and the choice of this proposal is key for achieving a high performance. In adaptive IS (AIS) methods, a set of proposals is iteratively improved. AIS is a relevant and timely m…
▽ More
Importance sampling (IS) is a powerful Monte Carlo (MC) methodology for approximating integrals, for instance in the context of Bayesian inference. In IS, the samples are simulated from the so-called proposal distribution, and the choice of this proposal is key for achieving a high performance. In adaptive IS (AIS) methods, a set of proposals is iteratively improved. AIS is a relevant and timely methodology although many limitations remain yet to be overcome, e.g., the curse of dimensionality in high-dimensional and multi-modal problems. Moreover, the Hamiltonian Monte Carlo (HMC) algorithm has become increasingly popular in machine learning and statistics. HMC has several appealing features such as its exploratory behavior, especially in high-dimensional targets, when other methods suffer. In this paper, we introduce the novel Hamiltonian adaptive importance sampling (HAIS) method. HAIS implements a two-step adaptive process with parallel HMC chains that cooperate at each iteration. The proposed HAIS efficiently adapts a population of proposals, extracting the advantages of HMC. HAIS can be understood as a particular instance of the generic layered AIS family with an additional resampling step. HAIS achieves a significant performance improvement in high-dimensional problems w.r.t. state-of-the-art algorithms. We discuss the statistical properties of HAIS and show its high performance in two challenging examples.
△ Less
Submitted 27 September, 2022;
originally announced September 2022.
-
Deep Learning Applications for Lung Cancer Diagnosis: A systematic review
Authors:
Hesamoddin Hosseini,
Reza Monsefi,
Shabnam Shadroo
Abstract:
Lung cancer has been one of the most prevalent disease in recent years. According to the research of this field, more than 200,000 cases are identified each year in the US. Uncontrolled multiplication and growth of the lung cells result in malignant tumour formation. Recently, deep learning algorithms, especially Convolutional Neural Networks (CNN), have become a superior way to automatically diag…
▽ More
Lung cancer has been one of the most prevalent disease in recent years. According to the research of this field, more than 200,000 cases are identified each year in the US. Uncontrolled multiplication and growth of the lung cells result in malignant tumour formation. Recently, deep learning algorithms, especially Convolutional Neural Networks (CNN), have become a superior way to automatically diagnose disease. The purpose of this article is to review different models that lead to different accuracy and sensitivity in the diagnosis of early-stage lung cancer and to help physicians and researchers in this field. The main purpose of this work is to identify the challenges that exist in lung cancer based on deep learning. The survey is systematically written that combines regular mapping and literature review to review 32 conference and journal articles in the field from 2016 to 2021. After analysing and reviewing the articles, the questions raised in the articles are being answered. This research is superior to other review articles in this field due to the complete review of relevant articles and systematic write up.
△ Less
Submitted 1 January, 2022;
originally announced January 2022.
-
Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifolds
Authors:
Reza Godaz,
Benyamin Ghojogh,
Reshad Hosseini,
Reza Monsefi,
Fakhri Karray,
Mark Crowley
Abstract:
This work concentrates on optimization on Riemannian manifolds. The Limited-memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) algorithm is a commonly used quasi-Newton method for numerical optimization in Euclidean spaces. Riemannian LBFGS (RLBFGS) is an extension of this method to Riemannian manifolds. RLBFGS involves computationally expensive vector transports as well as unfolding recursions using…
▽ More
This work concentrates on optimization on Riemannian manifolds. The Limited-memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) algorithm is a commonly used quasi-Newton method for numerical optimization in Euclidean spaces. Riemannian LBFGS (RLBFGS) is an extension of this method to Riemannian manifolds. RLBFGS involves computationally expensive vector transports as well as unfolding recursions using adjoint vector transports. In this article, we propose two mappings in the tangent space using the inverse second root and Cholesky decomposition. These mappings make both vector transport and adjoint vector transport identity and therefore isometric. Identity vector transport makes RLBFGS less computationally expensive and its isometry is also very useful in convergence analysis of RLBFGS. Moreover, under the proposed mappings, the Riemannian metric reduces to Euclidean inner product, which is much less computationally expensive. We focus on the Symmetric Positive Definite (SPD) manifolds which are beneficial in various fields such as data science and statistics. This work opens a research opportunity for extension of the proposed mappings to other well-known manifolds.
△ Less
Submitted 3 October, 2021; v1 submitted 24 August, 2021;
originally announced August 2021.
-
A Framework to Enhance Generalization of Deep Metric Learning methods using General Discriminative Feature Learning and Class Adversarial Neural Networks
Authors:
Karrar Al-Kaabi,
Reza Monsefi,
Davood Zabihzadeh
Abstract:
Metric learning algorithms aim to learn a distance function that brings the semantically similar data items together and keeps dissimilar ones at a distance. The traditional Mahalanobis distance learning is equivalent to find a linear projection. In contrast, Deep Metric Learning (DML) methods are proposed that automatically extract features from data and learn a non-linear transformation from inp…
▽ More
Metric learning algorithms aim to learn a distance function that brings the semantically similar data items together and keeps dissimilar ones at a distance. The traditional Mahalanobis distance learning is equivalent to find a linear projection. In contrast, Deep Metric Learning (DML) methods are proposed that automatically extract features from data and learn a non-linear transformation from input space to a semantically embedding space. Recently, many DML methods are proposed focused to enhance the discrimination power of the learned metric by providing novel sampling strategies or loss functions. This approach is very helpful when both the training and test examples are coming from the same set of categories. However, it is less effective in many applications of DML such as image retrieval and person-reidentification. Here, the DML should learn general semantic concepts from observed classes and employ them to rank or identify objects from unseen categories. Neglecting the generalization ability of the learned representation and just emphasizing to learn a more discriminative embedding on the observed classes may lead to the overfitting problem. To address this limitation, we propose a framework to enhance the generalization power of existing DML methods in a Zero-Shot Learning (ZSL) setting by general yet discriminative representation learning and employing a class adversarial neural network. To learn a more general representation, we propose to employ feature maps of intermediate layers in a deep neural network and enhance their discrimination power through an attention mechanism. Besides, a class adversarial network is utilized to enforce the deep model to seek class invariant features for the DML task. We evaluate our work on widely used machine vision datasets in a ZSL setting.
△ Less
Submitted 11 June, 2021;
originally announced June 2021.
-
Accurate and fast matrix factorization for low-rank learning
Authors:
Reza Godaz,
Reza Monsefi,
Faezeh Toutounian,
Reshad Hosseini
Abstract:
In this paper, we tackle two important problems in low-rank learning, which are partial singular value decomposition and numerical rank estimation of huge matrices. By using the concepts of Krylov subspaces such as Golub-Kahan bidiagonalization (GK-bidiagonalization) as well as Ritz vectors, we propose two methods for solving these problems in a fast and accurate way. Our experiments show the adva…
▽ More
In this paper, we tackle two important problems in low-rank learning, which are partial singular value decomposition and numerical rank estimation of huge matrices. By using the concepts of Krylov subspaces such as Golub-Kahan bidiagonalization (GK-bidiagonalization) as well as Ritz vectors, we propose two methods for solving these problems in a fast and accurate way. Our experiments show the advantages of the proposed methods compared to the traditional and randomized singular value decomposition methods. The proposed methods are appropriate for applications involving huge matrices where the accuracy of the desired singular values and also all of their corresponding singular vectors are essential. As a real application, we evaluate the performance of our methods on the problem of Riemannian similarity learning between two various image datasets of MNIST and USPS.
△ Less
Submitted 4 September, 2021; v1 submitted 21 April, 2021;
originally announced April 2021.
-
Neural Generalization of Multiple Kernel Learning
Authors:
Ahmad Navid Ghanizadeh,
Kamaledin Ghiasi-Shirazi,
Reza Monsefi,
Mohammadreza Qaraei
Abstract:
Multiple Kernel Learning is a conventional way to learn the kernel function in kernel-based methods. MKL algorithms enhance the performance of kernel methods. However, these methods have a lower complexity compared to deep learning models and are inferior to these models in terms of recognition accuracy. Deep learning models can learn complex functions by applying nonlinear transformations to data…
▽ More
Multiple Kernel Learning is a conventional way to learn the kernel function in kernel-based methods. MKL algorithms enhance the performance of kernel methods. However, these methods have a lower complexity compared to deep learning models and are inferior to these models in terms of recognition accuracy. Deep learning models can learn complex functions by applying nonlinear transformations to data through several layers. In this paper, we show that a typical MKL algorithm can be interpreted as a one-layer neural network with linear activation functions. By this interpretation, we propose a Neural Generalization of Multiple Kernel Learning (NGMKL), which extends the conventional multiple kernel learning framework to a multi-layer neural network with nonlinear activation functions. Our experiments on several benchmarks show that the proposed method improves the complexity of MKL algorithms and leads to higher recognition accuracy.
△ Less
Submitted 4 May, 2023; v1 submitted 26 February, 2021;
originally announced February 2021.
-
RELF: Robust Regression Extended with Ensemble Loss Function
Authors:
Hamideh Hajiabadi,
Reza Monsefi,
Hadi Sadoghi Yazdi
Abstract:
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regressor. We then propose a half-quadratic learning algorithm in order to find the p…
▽ More
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regressor. We then propose a half-quadratic learning algorithm in order to find the parameter of the regressor and the optimal weights associated with each loss function. Moreover, we show that our proposed loss function is robust in noisy environments. For a particular class of loss functions, we show that our proposed ensemble loss function is Bayes consistent and robust. Experimental evaluations on several datasets demonstrate that our proposed ensemble loss function significantly improves the performance of a simple regressor in comparison with state-of-the-art methods.
△ Less
Submitted 25 October, 2018;
originally announced October 2018.
-
TRLF: An Effective Semi-fragile Watermarking Method for Tamper Detection and Recovery based on LWT and FNN
Authors:
Behrouz Bolourian Haghighi,
Amir Hossein Taherinia,
Reza Monsefi
Abstract:
This paper proposes a novel method for tamper detection and recovery using semi-fragile data hiding, based on Lifting Wavelet Transform (LWT) and Feed-Forward Neural Network (FNN). In TRLF, first, the host image is decomposed up to one level using LWT, and the Discrete Cosine Transform (DCT) is applied to each 2*2 blocks of diagonal details. Next, a random binary sequence is embedded in each block…
▽ More
This paper proposes a novel method for tamper detection and recovery using semi-fragile data hiding, based on Lifting Wavelet Transform (LWT) and Feed-Forward Neural Network (FNN). In TRLF, first, the host image is decomposed up to one level using LWT, and the Discrete Cosine Transform (DCT) is applied to each 2*2 blocks of diagonal details. Next, a random binary sequence is embedded in each block as the watermark by correlating $DC$ coefficients. In authentication stage, first, the watermarked image geometry is reconstructed by using Speeded Up Robust Features (SURF) algorithm and extract watermark bits by using FNN. Afterward, logical exclusive-or operation between original and extracted watermark is applied to detect tampered region. Eventually, in the recovery stage, tampered regions are recovered by image digest which is generated by inverse halftoning technique. The performance and efficiency of TRLF and its robustness against various geometric, non-geometric and hybrid attacks are reported. From the experimental results, it can be seen that TRLF is superior in terms of robustness and quality of the digest and watermarked image respectively, compared to the-state-of-the-art fragile and semi-fragile watermarking methods. In addition, imperceptibility has been improved by using different correlation steps as the gain factor for flat (smooth) and texture (rough) blocks.
△ Less
Submitted 18 February, 2018;
originally announced February 2018.
-
On Extending Neural Networks with Loss Ensembles for Text Classification
Authors:
Hamideh Hajiabadi,
Diego Molla-Aliod,
Reza Monsefi
Abstract:
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta learning framework, ensemble techniques can easily be applied to many machine learning techniques. In this paper we propose a neural network extended with an ensemble loss function for text classification. The weight of each weak loss function is tuned within the training phase through…
▽ More
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta learning framework, ensemble techniques can easily be applied to many machine learning techniques. In this paper we propose a neural network extended with an ensemble loss function for text classification. The weight of each weak loss function is tuned within the training phase through the gradient propagation optimization method of the neural network. The approach is evaluated on several text classification datasets. We also evaluate its performance in various environments with several degrees of label noise. Experimental results indicate an improvement of the results and strong resilience against label noise in comparison with other methods.
△ Less
Submitted 14 November, 2017;
originally announced November 2017.
