-
A Comprehensive Survey on Knowledge Distillation
Authors:
Amir M. Mansourian,
Rozhan Ahmadi,
Masoud Ghafouri,
Amir Mohammad Babaei,
Elaheh Badali Golezani,
Zeynab Yasamani Ghamchi,
Vida Ramezanian,
Alireza Taherian,
Kimia Dinashi,
Amirali Miri,
Shohreh Kasaei
Abstract:
Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption.…
▽ More
Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption. This is especially concerning with the recent large-scale foundation models, Vision-Language Models (VLMs), and Large Language Models (LLMs). Knowledge Distillation (KD) is one of the prominent techniques proposed to address the aforementioned problems using a teacher-student architecture. More specifically, a lightweight student model is trained using additional knowledge from a cumbersome teacher model. In this work, a comprehensive survey of knowledge distillation methods is proposed. This includes reviewing KD from different aspects: distillation sources, distillation schemes, distillation algorithms, distillation by modalities, applications of distillation, and comparison among existing methods. In contrast to most existing surveys, which are either outdated or simply update former surveys, this work proposes a comprehensive survey with a new point of view and representation structure that categorizes and investigates the most recent methods in knowledge distillation. This survey considers various critically important subcategories, including KD for diffusion models, 3D inputs, foundational models, transformers, and LLMs. Furthermore, existing challenges in KD and possible future research directions are discussed. Github page of the project: https://github.com/IPL-Sharif/KD_Survey
△ Less
Submitted 15 March, 2025;
originally announced March 2025.
-
Adversarial Explainability: Utilizing Explainable Machine Learning in Bypassing IoT Botnet Detection Systems
Authors:
Mohammed M. Alani,
Atefeh Mashatan,
Ali Miri
Abstract:
Botnet detection based on machine learning have witnessed significant leaps in recent years, with the availability of large and reliable datasets that are extracted from real-life scenarios. Consequently, adversarial attacks on machine learning-based cybersecurity systems are posing a significant threat to the practicality of these solutions. In this paper, we introduce a novel attack that utilize…
▽ More
Botnet detection based on machine learning have witnessed significant leaps in recent years, with the availability of large and reliable datasets that are extracted from real-life scenarios. Consequently, adversarial attacks on machine learning-based cybersecurity systems are posing a significant threat to the practicality of these solutions. In this paper, we introduce a novel attack that utilizes machine learning model's explainability in evading detection by botnet detection systems. The proposed attack utilizes information obtained from model's explainability to build adversarial samples that can evade detection in a blackbox setting. The proposed attack was tested on a trained IoT botnet detection systems and was capable of bypassing the botnet detection with 0% detection by altering one feature only to generate the adversarial samples.
△ Less
Submitted 29 September, 2023;
originally announced October 2023.
-
A Model-Driven-Engineering Approach for Detecting Privilege Escalation in IoT Systems
Authors:
Atheer Abu Zaid,
Manar H. Alalfi,
Ali Miri
Abstract:
Software vulnerabilities in access control models can represent a serious threat in a system. In fact, OWASP lists broken access control as number 5 in severity among the top 10 vulnerabilities. In this paper, we study the permission model of an emerging Smart-Home platform, SmartThings, and explore an approach that detects privilege escalation in its permission model. Our approach is based on Mod…
▽ More
Software vulnerabilities in access control models can represent a serious threat in a system. In fact, OWASP lists broken access control as number 5 in severity among the top 10 vulnerabilities. In this paper, we study the permission model of an emerging Smart-Home platform, SmartThings, and explore an approach that detects privilege escalation in its permission model. Our approach is based on Model Driven Engineering (MDE) in addition to static analysis. This approach allows for better coverage of privilege escalation detection than static analysis alone, and takes advantage of analyzing free-form text that carries extra permissions details. Our experimental results demonstrate a very high accuracy for detecting over-privilege vulnerabilities in IoT applications
△ Less
Submitted 23 May, 2022;
originally announced May 2022.
-
On Defending Against Label Flipping Attacks on Malware Detection Systems
Authors:
Rahim Taheri,
Reza Javidan,
Mohammad Shojafar,
Zahra Pooranian,
Ali Miri,
Mauro Conti
Abstract:
Label manipulation attacks are a subclass of data poisoning attacks in adversarial machine learning used against different applications, such as malware detection. These types of attacks represent a serious threat to detection systems in environments having high noise rate or uncertainty, such as complex networks and Internet of Thing (IoT). Recent work in the literature has suggested using the…
▽ More
Label manipulation attacks are a subclass of data poisoning attacks in adversarial machine learning used against different applications, such as malware detection. These types of attacks represent a serious threat to detection systems in environments having high noise rate or uncertainty, such as complex networks and Internet of Thing (IoT). Recent work in the literature has suggested using the $K$-Nearest Neighboring (KNN) algorithm to defend against such attacks. However, such an approach can suffer from low to wrong detection accuracy. In this paper, we design an architecture to tackle the Android malware detection problem in IoT systems. We develop an attack mechanism based on Silhouette clustering method, modified for mobile Android platforms. We proposed two Convolutional Neural Network (CNN)-type deep learning algorithms against this \emph{Silhouette Clustering-based Label Flipping Attack (SCLFA)}. We show the effectiveness of these two defense algorithms - \emph{Label-based Semi-supervised Defense (LSD)} and \emph{clustering-based Semi-supervised Defense (CSD)} - in correcting labels being attacked. We evaluate the performance of the proposed algorithms by varying the various machine learning parameters on three Android datasets: Drebin, Contagio, and Genome and three types of features: API, intent, and permission. Our evaluation shows that using random forest feature selection and varying ratios of features can result in an improvement of up to 19\% accuracy when compared with the state-of-the-art method in the literature.
△ Less
Submitted 16 June, 2020; v1 submitted 12 August, 2019;
originally announced August 2019.
