-
Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures
Authors:
Evangelos Georganas,
Dhiraj Kalamkar,
Kirill Voronin,
Abhisek Kundu,
Antonio Noack,
Hans Pabst,
Alexander Breuer,
Alexander Heinecke
Abstract:
During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the programming methodology of DL and HPC systems is stagnant, relying on highly-optimized, yet platform-specific and inflexible vendor-optimized libraries. Such libraries provide close-to-peak performance on specific platforms…
▽ More
During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the programming methodology of DL and HPC systems is stagnant, relying on highly-optimized, yet platform-specific and inflexible vendor-optimized libraries. Such libraries provide close-to-peak performance on specific platforms, kernels and shapes thereof that vendors have dedicated optimizations efforts, while they underperform in the remaining use-cases, yielding non-portable codes with performance glass-jaws. This work introduces a framework to develop efficient, portable DL and HPC kernels for modern CPU architectures. We decompose the kernel development in two steps: 1) Expressing the computational core using Tensor Processing Primitives (TPPs): a compact, versatile set of 2D-tensor operators, 2) Expressing the logical loops around TPPs in a high-level, declarative fashion whereas the exact instantiation (ordering, tiling, parallelization) is determined via simple knobs. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
△ Less
Submitted 15 March, 2024; v1 submitted 25 April, 2023;
originally announced April 2023.
-
TCAB: A Large-Scale Text Classification Attack Benchmark
Authors:
Kalyani Asthana,
Zhouhang Xie,
Wencong You,
Adam Noack,
Jonathan Brophy,
Sameer Singh,
Daniel Lowd
Abstract:
We introduce the Text Classification Attack Benchmark (TCAB), a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attacks targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. Unlike standard text classific…
▽ More
We introduce the Text Classification Attack Benchmark (TCAB), a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attacks targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. Unlike standard text classification, text attacks must be understood in the context of the target classifier that is being attacked, and thus features of the target classifier are important as well. TCAB includes all attack instances that are successful in flipping the predicted label; a subset of the attacks are also labeled by human annotators to determine how frequently the primary semantics are preserved. The process of generating attacks is automated, so that TCAB can easily be extended to incorporate new text attacks and better classifiers as they are developed. In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization. TCAB code and dataset are available at https://react-nlp.github.io/tcab/.
△ Less
Submitted 21 October, 2022;
originally announced October 2022.
-
Identifying Adversarial Attacks on Text Classifiers
Authors:
Zhouhang Xie,
Jonathan Brophy,
Adam Noack,
Wencong You,
Kalyani Asthana,
Carter Perkins,
Sabrina Reis,
Sameer Singh,
Daniel Lowd
Abstract:
The landscape of adversarial attacks against text classifiers continues to grow, with new attacks developed every year and many of them available in standard toolkits, such as TextAttack and OpenAttack. In response, there is a growing body of work on robust learning, which reduces vulnerability to these attacks, though sometimes at a high cost in compute time or accuracy. In this paper, we take an…
▽ More
The landscape of adversarial attacks against text classifiers continues to grow, with new attacks developed every year and many of them available in standard toolkits, such as TextAttack and OpenAttack. In response, there is a growing body of work on robust learning, which reduces vulnerability to these attacks, though sometimes at a high cost in compute time or accuracy. In this paper, we take an alternate approach -- we attempt to understand the attacker by analyzing adversarial text to determine which methods were used to create it. Our first contribution is an extensive dataset for attack detection and labeling: 1.5~million attack instances, generated by twelve adversarial attacks targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. As our second contribution, we use this dataset to develop and benchmark a number of classifiers for attack identification -- determining if a given text has been adversarially manipulated and by which attack. As a third contribution, we demonstrate the effectiveness of three classes of features for these tasks: text properties, capturing content and presentation of text; language model properties, determining which tokens are more or less probable throughout the input; and target model properties, representing how the text classifier is influenced by the attack, including internal node activations. Overall, this represents a first step towards forensics for adversarial attacks against text classifiers.
△ Less
Submitted 21 January, 2022;
originally announced January 2022.
-
An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
Authors:
Adam Noack,
Isaac Ahern,
Dejing Dou,
Boyang Li
Abstract:
Deep neural networks (DNNs) have had many successes, but they suffer from two major issues: (1) a vulnerability to adversarial examples and (2) a tendency to elude human interpretation. Interestingly, recent empirical and theoretical evidence suggests these two seemingly disparate issues are actually connected. In particular, robust models tend to provide more interpretable gradients than non-robu…
▽ More
Deep neural networks (DNNs) have had many successes, but they suffer from two major issues: (1) a vulnerability to adversarial examples and (2) a tendency to elude human interpretation. Interestingly, recent empirical and theoretical evidence suggests these two seemingly disparate issues are actually connected. In particular, robust models tend to provide more interpretable gradients than non-robust models. However, whether this relationship works in the opposite direction remains obscure. With this paper, we seek empirical answers to the following question: can models acquire adversarial robustness when they are trained to have interpretable gradients? We introduce a theoretically inspired technique called Interpretation Regularization (IR), which encourages a model's gradients to (1) match the direction of interpretable target salience maps and (2) have small magnitude. To assess model performance and tease apart factors that contribute to adversarial robustness, we conduct extensive experiments on MNIST and CIFAR-10 with both $\ell_2$ and $\ell_\infty$ attacks. We demonstrate that training the networks to have interpretable gradients improves their robustness to adversarial perturbations. Applying the network interpretation technique SmoothGrad yields additional performance gains, especially in cross-norm attacks and under heavy perturbations. The results indicate that the interpretability of the model gradients is a crucial factor for adversarial robustness. Code for the experiments can be found at https://github.com/a1noack/interp_regularization.
△ Less
Submitted 3 December, 2020; v1 submitted 6 December, 2019;
originally announced December 2019.
-
NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
Authors:
Isaac Ahern,
Adam Noack,
Luis Guzman-Nateras,
Dejing Dou,
Boyang Li,
Jun Huan
Abstract:
The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold and SP-LIME ag…
▽ More
The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, NormLIME for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by NormLIME to other feature importance metrics. Numerical experiments confirm that NormLIME is effective at recognizing important features.
△ Less
Submitted 15 October, 2019; v1 submitted 9 September, 2019;
originally announced September 2019.
-
Fast computation of the principal components of genotype matrices in Julia
Authors:
Jiahao Chen,
Andreas Noack,
Alan Edelman
Abstract:
Finding the largest few principal components of a matrix of genetic data is a common task in genome-wide association studies (GWASs), both for dimensionality reduction and for identifying unwanted factors of variation. We describe a simple random matrix model for matrices that arise in GWASs, showing that the singular values have a bulk behavior that obeys a Marchenko-Pastur distributed with a han…
▽ More
Finding the largest few principal components of a matrix of genetic data is a common task in genome-wide association studies (GWASs), both for dimensionality reduction and for identifying unwanted factors of variation. We describe a simple random matrix model for matrices that arise in GWASs, showing that the singular values have a bulk behavior that obeys a Marchenko-Pastur distributed with a handful of large outliers. We also implement Golub-Kahan-Lanczos (GKL) bidiagonalization in the Julia programming language, providing thick restarting and a choice between full and partial reorthogonalization strategies to control numerical roundoff. Our implementation of GKL bidiagonalization is up to 36 times faster than software tools used commonly in genomics data analysis for computing principal components, such as EIGENSOFT and FlashPCA, which use dense LAPACK routines and randomized subspace iteration respectively.
△ Less
Submitted 9 August, 2018;
originally announced August 2018.
-
Cataloging the Visible Universe through Bayesian Inference at Petascale
Authors:
Jeffrey Regier,
Kiran Pamnany,
Keno Fischer,
Andreas Noack,
Maximilian Lam,
Jarrett Revels,
Steve Howard,
Ryan Giordano,
David Schlegel,
Jon McAuliffe,
Rollin Thomas,
Prabhat
Abstract:
Astronomical catalogs derived from wide-field imaging surveys are an important tool for understanding the Universe. We construct an astronomical catalog from 55 TB of imaging data using Celeste, a Bayesian variational inference code written entirely in the high-productivity programming language Julia. Using over 1.3 million threads on 650,000 Intel Xeon Phi cores of the Cori Phase II supercomputer…
▽ More
Astronomical catalogs derived from wide-field imaging surveys are an important tool for understanding the Universe. We construct an astronomical catalog from 55 TB of imaging data using Celeste, a Bayesian variational inference code written entirely in the high-productivity programming language Julia. Using over 1.3 million threads on 650,000 Intel Xeon Phi cores of the Cori Phase II supercomputer, Celeste achieves a peak rate of 1.54 DP PFLOP/s. Celeste is able to jointly optimize parameters for 188M stars and galaxies, loading and processing 178 TB across 8192 nodes in 14.6 minutes. To achieve this, Celeste exploits parallelism at multiple levels (cluster, node, and thread) and accelerates I/O through Cori's Burst Buffer. Julia's native performance enables Celeste to employ high-level constructs without resorting to hand-written or generated low-level code (C/C++/Fortran), and yet achieve petascale performance.
△ Less
Submitted 30 January, 2018;
originally announced January 2018.
-
Game-Theoretic Model of Incentivizing Privacy-Aware Users to Consent to Location Tracking
Authors:
Emmanouil Panaousis,
Aron Laszka,
Johannes Pohl,
Andreas Noack,
Tansu Alpcan
Abstract:
Nowadays, mobile users have a vast number of applications and services at their disposal. Each of these might impose some privacy threats on users' "Personally Identifiable Information" (PII). Location privacy is a crucial part of PII, and as such, privacy-aware users wish to maximize it. This privacy can be, for instance, threatened by a company, which collects users' traces and shares them with…
▽ More
Nowadays, mobile users have a vast number of applications and services at their disposal. Each of these might impose some privacy threats on users' "Personally Identifiable Information" (PII). Location privacy is a crucial part of PII, and as such, privacy-aware users wish to maximize it. This privacy can be, for instance, threatened by a company, which collects users' traces and shares them with third parties. To maximize their location privacy, users can decide to get offline so that the company cannot localize their devices. The longer a user stays connected to a network, the more services he might receive, but his location privacy decreases. In this paper, we analyze the trade-off between location privacy, the level of services that a user experiences, and the profit of the company. To this end, we formulate a Stackelberg Bayesian game between the User (follower) and the Company (leader). We present theoretical results characterizing the equilibria of the game. To the best of our knowledge, our work is the first to model the economically rational decision-making of the service provider (i.e., the Company) in conjunction with the rational decision-making of users who wish to protect their location privacy. To evaluate the performance of our approach, we have used real-data from a testbed, and we have also shown that the game-theoretic strategy of the Company outperforms non-strategic methods. Finally, we have considered different User privacy types, and have determined the service level that incentivizes the User to stay connected as long as possible.
△ Less
Submitted 2 January, 2016;
originally announced January 2016.
-
Multi-level algorithms for modularity clustering
Authors:
Andreas Noack,
Randolf Rotta
Abstract:
Modularity is one of the most widely used quality measures for graph clusterings. Maximizing modularity is NP-hard, and the runtime of exact algorithms is prohibitive for large graphs. A simple and effective class of heuristics coarsens the graph by iteratively merging clusters (starting from singletons), and optionally refines the resulting clustering by iteratively moving individual vertices b…
▽ More
Modularity is one of the most widely used quality measures for graph clusterings. Maximizing modularity is NP-hard, and the runtime of exact algorithms is prohibitive for large graphs. A simple and effective class of heuristics coarsens the graph by iteratively merging clusters (starting from singletons), and optionally refines the resulting clustering by iteratively moving individual vertices between clusters. Several heuristics of this type have been proposed in the literature, but little is known about their relative performance.
This paper experimentally compares existing and new coarsening- and refinement-based heuristics with respect to their effectiveness (achieved modularity) and efficiency (runtime). Concerning coarsening, it turns out that the most widely used criterion for merging clusters (modularity increase) is outperformed by other simple criteria, and that a recent algorithm by Schuetz and Caflisch is no improvement over simple greedy coarsening for these criteria. Concerning refinement, a new multi-level algorithm is shown to produce significantly better clusterings than conventional single-level algorithms. A comparison with published benchmark results and algorithm implementations shows that combinations of coarsening and multi-level refinement are competitive with the best algorithms in the literature.
△ Less
Submitted 29 December, 2008; v1 submitted 22 December, 2008;
originally announced December 2008.
-
Modularity clustering is force-directed layout
Authors:
Andreas Noack
Abstract:
Two natural and widely used representations for the community structure of networks are clusterings, which partition the vertex set into disjoint subsets, and layouts, which assign the vertices to positions in a metric space. This paper unifies prominent characterizations of layout quality and clustering quality, by showing that energy models of pairwise attraction and repulsion subsume Newman a…
▽ More
Two natural and widely used representations for the community structure of networks are clusterings, which partition the vertex set into disjoint subsets, and layouts, which assign the vertices to positions in a metric space. This paper unifies prominent characterizations of layout quality and clustering quality, by showing that energy models of pairwise attraction and repulsion subsume Newman and Girvan's modularity measure. Layouts with optimal energy are relaxations of, and are thus consistent with, clusterings with optimal modularity, which is of practical relevance because both representations are complementary and often used together.
△ Less
Submitted 9 December, 2008; v1 submitted 25 July, 2008;
originally announced July 2008.
-
CrocoPat 2.1 Introduction and Reference Manual
Authors:
Dirk Beyer,
Andreas Noack
Abstract:
CrocoPat is an efficient, powerful and easy-to-use tool for manipulating relations of arbitrary arity, including directed graphs. This manual provides an introduction to and a reference for CrocoPat and its programming language RML. It includes several application examples, in particular from the analysis of structural models of software systems.
CrocoPat is an efficient, powerful and easy-to-use tool for manipulating relations of arbitrary arity, including directed graphs. This manual provides an introduction to and a reference for CrocoPat and its programming language RML. It includes several application examples, in particular from the analysis of structural models of software systems.
△ Less
Submitted 7 September, 2004;
originally announced September 2004.
