-
Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions
Authors:
Satyaki Das,
Syeda Tasnim Fabiha,
Saad Shafiq,
Nenad Medvidovic
Abstract:
Recent research has revealed that the reported results of an emerging body of DL-based techniques for detecting software vulnerabilities are not reproducible, either across different datasets or on unseen samples. This paper aims to provide the foundation for properly evaluating the research in this domain. We do so by analyzing prior work and existing vulnerability datasets for the syntactic and…
▽ More
Recent research has revealed that the reported results of an emerging body of DL-based techniques for detecting software vulnerabilities are not reproducible, either across different datasets or on unseen samples. This paper aims to provide the foundation for properly evaluating the research in this domain. We do so by analyzing prior work and existing vulnerability datasets for the syntactic and semantic features of code that contribute to vulnerability, as well as features that falsely correlate with vulnerability. We provide a novel, uniform representation to capture both sets of features, and use this representation to detect the presence of both vulnerability and spurious features in code. To this end, we design two types of code perturbations: feature preserving perturbations (FPP) ensure that the vulnerability feature remains in a given code sample, while feature eliminating perturbations (FEP) eliminate the feature from the code sample. These perturbations aim to measure the influence of spurious and vulnerability features on the predictions of a given vulnerability detection solution. To evaluate how the two classes of perturbations influence predictions, we conducted a large-scale empirical study on five state-of-the-art DL-based vulnerability detectors. Our study shows that, for vulnerability features, only ~2% of FPPs yield the undesirable effect of a prediction changing among the five detectors on average. However, on average, ~84% of FEPs yield the undesirable effect of retaining the vulnerability predictions. For spurious features, we observed that FPPs yielded a drop in recall up to 29% for graph-based detectors. We present the reasons underlying these results and suggest strategies for improving DNN-based vulnerability detectors. We provide our perturbation-based evaluation framework as a public resource to enable independent future evaluation of vulnerability detectors.
△ Less
Submitted 23 May, 2025; v1 submitted 22 January, 2025;
originally announced January 2025.
-
Improving DNN Modularization via Activation-Driven Training
Authors:
Tuan Ngo,
Abid Hassan,
Saad Shafiq,
Nenad Medvidovic
Abstract:
Deep Neural Networks (DNNs) suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses acr…
▽ More
Deep Neural Networks (DNNs) suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity. In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and three datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 29% less training time; (2) the resultant modules generated by MODA comprise 2.4x fewer weights and 3.5x less weight overlap while (3) preserving the original model's accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.
△ Less
Submitted 1 November, 2024;
originally announced November 2024.
-
Toward Improved Deep Learning-based Vulnerability Detection
Authors:
Adriana Sejfia,
Satyaki Das,
Saad Shafiq,
Nenad Medvidović
Abstract:
Deep learning (DL) has been a common thread across several recent techniques for vulnerability detection. The rise of large, publicly available datasets of vulnerabilities has fueled the learning process underpinning these techniques. While these datasets help the DL-based vulnerability detectors, they also constrain these detectors' predictive abilities. Vulnerabilities in these datasets have to…
▽ More
Deep learning (DL) has been a common thread across several recent techniques for vulnerability detection. The rise of large, publicly available datasets of vulnerabilities has fueled the learning process underpinning these techniques. While these datasets help the DL-based vulnerability detectors, they also constrain these detectors' predictive abilities. Vulnerabilities in these datasets have to be represented in a certain way, e.g., code lines, functions, or program slices within which the vulnerabilities exist. We refer to this representation as a base unit. The detectors learn how base units can be vulnerable and then predict whether other base units are vulnerable. We have hypothesized that this focus on individual base units harms the ability of the detectors to properly detect those vulnerabilities that span multiple base units (or MBU vulnerabilities). For vulnerabilities such as these, a correct detection occurs when all comprising base units are detected as vulnerable. Verifying how existing techniques perform in detecting all parts of a vulnerability is important to establish their effectiveness for other downstream tasks. To evaluate our hypothesis, we conducted a study focusing on three prominent DL-based detectors: ReVeal, DeepWukong, and LineVul. Our study shows that all three detectors contain MBU vulnerabilities in their respective datasets. Further, we observed significant accuracy drops when detecting these types of vulnerabilities. We present our study and a framework that can be used to help DL-based detectors toward the proper inclusion of MBU vulnerabilities.
△ Less
Submitted 24 January, 2025; v1 submitted 5 March, 2024;
originally announced March 2024.
-
Balanced Knowledge Distribution among Software Development Teams -- Observations from Open-Source and Closed-Source Software Development
Authors:
Saad Shafiq,
Christoph Mayr-Dorn,
Atif Mashkoor,
Alexander Egyed
Abstract:
In software development teams, developer turnover is among the primary reasons for project failures as it leads to a great void of knowledge and strain for the newcomers. Unfortunately, no established methods exist to measure how knowledge is distributed among development teams. Knowing how this knowledge evolves and is owned by key developers in a project helps managers reduce risks caused by tur…
▽ More
In software development teams, developer turnover is among the primary reasons for project failures as it leads to a great void of knowledge and strain for the newcomers. Unfortunately, no established methods exist to measure how knowledge is distributed among development teams. Knowing how this knowledge evolves and is owned by key developers in a project helps managers reduce risks caused by turnover. To this end, this paper introduces a novel, realistic representation of domain knowledge distribution: the ConceptRealm. To construct the ConceptRealm, we employ a latent Dirichlet allocation model to represent textual features obtained from 300k issues and 1.3M comments from 518 open-source projects. We analyze whether the newly emerged issues and developers share similar concepts or how aligned the developers' concepts are with the team over time. We also investigate the impact of leaving members on the frequency of concepts. Finally, we evaluate the soundness of our approach to closed-source software, thus allowing the validation of the results from a practical standpoint. We find out that the ConceptRealm can represent the high-level domain knowledge within a team and can be utilized to predict the alignment of developers with issues. We also observe that projects exhibit many keepers independent of project maturity and that abruptly leaving keepers harm the team's concept familiarity.
△ Less
Submitted 26 July, 2022;
originally announced July 2022.
-
Charged particle dynamics in the surrounding of Schwarzschild anti-de Sitter black hole with topological defect immersed in an external magnetic field
Authors:
Sidra Shafiq,
Saqib Hussain,
Muhammad Ozair,
Adnan Aslam,
Takasar Hussain
Abstract:
In this paper, geodesic motion of the charged particles in the vicinity of event horizon of Schwarzschild anti-de-Sitter black hole (BH) with topological defects has been investigated. Weakly magnetized environment is considered in the surrounding of BH which only effects the motion of the particles and doesn't effect the geometry of the BH. Hence, particles are under the influence of gravity and…
▽ More
In this paper, geodesic motion of the charged particles in the vicinity of event horizon of Schwarzschild anti-de-Sitter black hole (BH) with topological defects has been investigated. Weakly magnetized environment is considered in the surrounding of BH which only effects the motion of the particles and doesn't effect the geometry of the BH. Hence, particles are under the influence of gravity and electromagnetic forces. We have explored the effect of magnetic field on the trajectories of the particles and more importantly on the position of the innermost stable circular orbit. It is observed that the trajectories of the particles in the surrounding of BH are chaotic. Escape conditions of the particles under the influence of gravitomagnetic force are also discussed. Moreover, the escape velocity of particles and its different features have been investigated in the presence and absence of magnetic field. Effect of dark energy on the size of event horizon, mass of the BH and stability of the orbits of the particles have also been explored in detail.
These studies can be used to estimate the power of relativistic jets originated from the vicinity of BH.
△ Less
Submitted 26 October, 2021;
originally announced October 2021.
-
TaskAllocator: A Recommendation Approach for Role-based Tasks Allocation in Agile Software Development
Authors:
Saad Shafiq,
Atif Mashkoor,
Christoph Mayr-Dorn,
Alexander Egyed
Abstract:
In this paper, we propose a recommendation approach -- TaskAllocator -- in order to predict the assignment of incoming tasks to potential befitting roles. The proposed approach, identifying team roles rather than individual persons, allows project managers to perform better tasks allocation in case the individual developers are over-utilized or moved on to different roles/projects. We evaluated ou…
▽ More
In this paper, we propose a recommendation approach -- TaskAllocator -- in order to predict the assignment of incoming tasks to potential befitting roles. The proposed approach, identifying team roles rather than individual persons, allows project managers to perform better tasks allocation in case the individual developers are over-utilized or moved on to different roles/projects. We evaluated our approach on ten agile case study projects obtained from the Taiga.io repository. In order to determine the TaskAllocator's performance, we have conducted a benchmark study by comparing it with contemporary machine learning models. The applicability of the TaskAllocator was assessed through a plugin that can be integrated with JIRA and provides recommendations about suitable roles whenever a new task is added to the project. Lastly, the source code of the plugin and the dataset employed have been made public.
△ Less
Submitted 3 March, 2021;
originally announced March 2021.
-
Blockchain-Federated-Learning and Deep Learning Models for COVID-19 detection using CT Imaging
Authors:
Rajesh Kumar,
Abdullah Aman Khan,
Sinmin Zhang,
Jay Kumar,
Ting Yang,
Noorbakhash Amiri Golalirz,
Zakria,
Ikram Ali,
Sidra Shafiq,
WenYong Wang
Abstract:
With the increase of COVID-19 cases worldwide, an effective way is required to diagnose COVID-19 patients. The primary problem in diagnosing COVID-19 patients is the shortage and reliability of testing kits, due to the quick spread of the virus, medical practitioners are facing difficulty identifying the positive cases. The second real-world problem is to share the data among the hospitals globall…
▽ More
With the increase of COVID-19 cases worldwide, an effective way is required to diagnose COVID-19 patients. The primary problem in diagnosing COVID-19 patients is the shortage and reliability of testing kits, due to the quick spread of the virus, medical practitioners are facing difficulty identifying the positive cases. The second real-world problem is to share the data among the hospitals globally while keeping in view the privacy concerns of the organizations. Building a collaborative model and preserving privacy are major concerns for training a global deep learning model. This paper proposes a framework that collects a small amount of data from different sources (various hospitals) and trains a global deep learning model using blockchain based federated learning. Blockchain technology authenticates the data and federated learning trains the model globally while preserving the privacy of the organization. First, we propose a data normalization technique that deals with the heterogeneity of data as the data is gathered from different hospitals having different kinds of CT scanners. Secondly, we use Capsule Network-based segmentation and classification to detect COVID-19 patients. Thirdly, we design a method that can collaboratively train a global model using blockchain technology with federated learning while preserving privacy. Additionally, we collected real-life COVID-19 patients data, which is, open to the research community. The proposed framework can utilize up-to-date data which improves the recognition of computed tomography (CT) images. Finally, our results demonstrate a better performance to detect COVID-19 patients.
△ Less
Submitted 8 December, 2020; v1 submitted 10 July, 2020;
originally announced July 2020.
-
Machine Learning for Software Engineering: A Systematic Mapping
Authors:
Saad Shafiq,
Atif Mashkoor,
Christoph Mayr-Dorn,
Alexander Egyed
Abstract:
Context: The software development industry is rapidly adopting machine learning for transitioning modern day software systems towards highly intelligent and self-learning systems. However, the full potential of machine learning for improving the software engineering life cycle itself is yet to be discovered, i.e., up to what extent machine learning can help reducing the effort/complexity of softwa…
▽ More
Context: The software development industry is rapidly adopting machine learning for transitioning modern day software systems towards highly intelligent and self-learning systems. However, the full potential of machine learning for improving the software engineering life cycle itself is yet to be discovered, i.e., up to what extent machine learning can help reducing the effort/complexity of software engineering and improving the quality of resulting software systems. To date, no comprehensive study exists that explores the current state-of-the-art on the adoption of machine learning across software engineering life cycle stages. Objective: This article addresses the aforementioned problem and aims to present a state-of-the-art on the growing number of uses of machine learning in software engineering. Method: We conduct a systematic mapping study on applications of machine learning to software engineering following the standard guidelines and principles of empirical software engineering. Results: This study introduces a machine learning for software engineering (MLSE) taxonomy classifying the state-of-the-art machine learning techniques according to their applicability to various software engineering life cycle stages. Overall, 227 articles were rigorously selected and analyzed as a result of this study. Conclusion: From the selected articles, we explore a variety of aspects that should be helpful to academics and practitioners alike in understanding the potential of adopting machine learning techniques during software engineering projects.
△ Less
Submitted 27 May, 2020;
originally announced May 2020.
-
Spin-orbit splittings in heavy-light mesons and Dirac equation
Authors:
Riazuddin,
Sidra Shafiq
Abstract:
The spin-orbit splitting in heavy-light mesons is seen to be suppressed experimentally. It is shown that it can be understood qualitatively in the frame work of Dirac theory. An alternative derivation of a relativistic dynamical symmetry for the Dirac Hamiltonian, which suppresses spin orbit splitting, is also given. However it is shown that such a symmetry is not needed since the spin-orbit split…
▽ More
The spin-orbit splitting in heavy-light mesons is seen to be suppressed experimentally. It is shown that it can be understood qualitatively in the frame work of Dirac theory. An alternative derivation of a relativistic dynamical symmetry for the Dirac Hamiltonian, which suppresses spin orbit splitting, is also given. However it is shown that such a symmetry is not needed since the spin-orbit splitting in Dirac theory with Coulomb like potential (as is the case for the one gluon exchange potential in pQCD) is small anyway.
△ Less
Submitted 7 October, 2011;
originally announced October 2011.
