Showing 1–38 of 38 results for author: Luckow, A

Typical Machine Learning Datasets as Low-Depth Quantum Circuits

Authors: Florian J. Kiwit, Bernhard Jobst, Andre Luckow, Frank Pollmann, Carlos A. Riofrío

Abstract: Quantum machine learning (QML) is an emerging field that investigates the capabilities of quantum computers for learning tasks. While QML models can theoretically offer advantages such as exponential speed-ups, challenges in data loading and the ability to scale to relevant problem sizes have prevented demonstrations of such advantages on practical problems. In particular, the encoding of arbitrar… ▽ More Quantum machine learning (QML) is an emerging field that investigates the capabilities of quantum computers for learning tasks. While QML models can theoretically offer advantages such as exponential speed-ups, challenges in data loading and the ability to scale to relevant problem sizes have prevented demonstrations of such advantages on practical problems. In particular, the encoding of arbitrary classical data into quantum states usually comes at a high computational cost, either in terms of qubits or gate count. However, real-world data typically exhibits some inherent structure (such as image data) which can be leveraged to load them with a much smaller cost on a quantum computer. This work further develops an efficient algorithm for finding low-depth quantum circuits to load classical image data as quantum states. To evaluate its effectiveness, we conduct systematic studies on the MNIST, Fashion-MNIST, CIFAR-10, and Imagenette datasets. The corresponding circuits for loading the full large-scale datasets are available publicly as PennyLane datasets and can be used by the community for their own benchmarks. We further analyze the performance of various quantum classifiers, such as quantum kernel methods, parameterized quantum circuits, and tensor-network classifiers, and we compare them to convolutional neural networks. In particular, we focus on the performance of the quantum classifiers as we introduce nonlinear functions of the input state, e.g., by letting the circuit parameters depend on the input state. △ Less

Submitted 6 May, 2025; originally announced May 2025.

Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools

Authors: Hyerim Park, Malin Eiband, Andre Luckow, Michael Sedlmair

Abstract: Generative AI (GenAI) tools are increasingly integrated into design workflows. While text prompts remain the primary input method for GenAI image tools, designers often struggle to craft effective ones. Moreover, research has primarily focused on input methods for ideation, with limited attention to refinement tasks. This study explores designers' preferences for three input methods - text prompts… ▽ More Generative AI (GenAI) tools are increasingly integrated into design workflows. While text prompts remain the primary input method for GenAI image tools, designers often struggle to craft effective ones. Moreover, research has primarily focused on input methods for ideation, with limited attention to refinement tasks. This study explores designers' preferences for three input methods - text prompts, annotations, and scribbles - through a preliminary digital paper-based study with seven professional designers. Designers preferred annotations for spatial adjustments and referencing in-image elements, while scribbles were favored for specifying attributes such as shape, size, and position, often combined with other methods. Text prompts excelled at providing detailed descriptions or when designers sought greater GenAI creativity. However, designers expressed concerns about AI misinterpreting annotations and scribbles and the effort needed to create effective text prompts. These insights inform GenAI interface design to better support refinement tasks, align with workflows, and enhance communication with AI systems. △ Less

Submitted 6 March, 2025; v1 submitted 5 March, 2025; originally announced March 2025.

Pilot-Quantum: A Quantum-HPC Middleware for Resource, Workload and Task Management

Authors: Pradeep Mantha, Florian J. Kiwit, Nishant Saurabh, Shantenu Jha, Andre Luckow

Abstract: As quantum hardware advances, integrating quantum processing units (QPUs) into HPC environments and managing diverse infrastructure and software stacks becomes increasingly essential. Pilot-Quantum addresses these challenges as a middleware designed to provide unified application-level management of resources and workloads across hybrid quantum-classical environments. It is built on a rigorous ana… ▽ More As quantum hardware advances, integrating quantum processing units (QPUs) into HPC environments and managing diverse infrastructure and software stacks becomes increasingly essential. Pilot-Quantum addresses these challenges as a middleware designed to provide unified application-level management of resources and workloads across hybrid quantum-classical environments. It is built on a rigorous analysis of existing quantum middleware systems and application execution patterns. It implements the Pilot Abstraction conceptual model, originally developed for HPC, to manage resources, workloads, and tasks. It is designed for quantum applications that rely on task parallelism, including (i) hybrid algorithms, such as variational approaches, and (ii) circuit cutting systems, used to partition and execute large quantum circuits. Pilot-Quantum facilitates seamless integration of QPUs, classical CPUs, and GPUs, while supporting high-level programming frameworks like Qiskit and Pennylane. This enables users to efficiently design and execute hybrid workflows across diverse computing resources. The capabilities of Pilot-Quantum are demonstrated through mini-apps -- simplified yet representative kernels focusing on critical performance bottlenecks. We demonstrate the capabilities of Pilot-Quantum through multiple mini-apps, including different circuit executions (e.g., using IBMś Eagle QPU and simulators), circuit cutting, and quantum machine learning scenarios. △ Less

Submitted 28 May, 2025; v1 submitted 24 December, 2024; originally announced December 2024.

Quantum Computing for Automotive Applications

Authors: Carlos A. Riofrío, Johannes Klepsch, Jernej Rudi Finžgar, Florian Kiwit, Leonhard Hölscher, Marvin Erdmann, Lukas Müller, Chandan Kumar, Youssef Achari Berrada, Andre Luckow

Abstract: Quantum computing could impact various industries, with the automotive industry with many computational challenges, from optimizing supply chains and manufacturing to vehicle engineering, being particularly promising. This chapter investigates state-of-the-art quantum algorithms to enhance efficiency, accuracy, and scalability across the automotive value chain. We explore recent advances in quantu… ▽ More Quantum computing could impact various industries, with the automotive industry with many computational challenges, from optimizing supply chains and manufacturing to vehicle engineering, being particularly promising. This chapter investigates state-of-the-art quantum algorithms to enhance efficiency, accuracy, and scalability across the automotive value chain. We explore recent advances in quantum optimization, machine learning, and numerical and chemistry simulations, highlighting their potential and limitations. We identify and discuss key challenges in near-term and fault-tolerant algorithms and their practical use in industrial applications. While quantum algorithms show potential in many application domains, current noisy intermediate-scale quantum hardware limits scale and, thus, business benefits. In the long term, fault-tolerant systems promise theoretical speedups; however, they also require further progress in hardware and software (e.g., related to error correction and data loading). We expect that with this progress, significant practical benefits will emerge eventually. △ Less

Submitted 24 March, 2025; v1 submitted 21 September, 2024; originally announced September 2024.

Assessing the Requirements for Industry Relevant Quantum Computation

Authors: Anna M. Krol, Marvin Erdmann, Ewan Munro, Andre Luckow, Zaid Al-Ars

Abstract: In this paper, we use open-source tools to perform quantum resource estimation to assess the requirements for industry-relevant quantum computation. Our analysis uses the problem of industrial shift scheduling in manufacturing and the Quantum Industrial Shift Scheduling algorithm. We base our figures of merit on current technology, as well as theoretical high-fidelity scenarios for superconducting… ▽ More In this paper, we use open-source tools to perform quantum resource estimation to assess the requirements for industry-relevant quantum computation. Our analysis uses the problem of industrial shift scheduling in manufacturing and the Quantum Industrial Shift Scheduling algorithm. We base our figures of merit on current technology, as well as theoretical high-fidelity scenarios for superconducting qubit platforms. We find that the execution time of gate and measurement operations determines the overall computational runtime more strongly than the system error rates. Moreover, achieving a quantum speedup would not only require low system error rates ($10^{-6}$ or better), but also measurement operations with an execution time below 10ns. This rules out the possibility of near-term quantum advantage for this use case, and suggests that significant technological or algorithmic progress will be needed before such an advantage can be achieved. △ Less

Submitted 5 August, 2024; originally announced August 2024.

Quantum-Inspired Fluid Simulation of 2D Turbulence with GPU Acceleration

Authors: Leonhard Hölscher, Pooja Rao, Lukas Müller, Johannes Klepsch, Andre Luckow, Tobias Stollenwerk, Frank K. Wilhelm

Abstract: Tensor network algorithms can efficiently simulate complex quantum many-body systems by utilizing knowledge of their structure and entanglement. These methodologies have been adapted recently for solving the Navier-Stokes equations, which describe a spectrum of fluid phenomena, from the aerodynamics of vehicles to weather patterns. Within this quantum-inspired paradigm, velocity is encoded as matr… ▽ More Tensor network algorithms can efficiently simulate complex quantum many-body systems by utilizing knowledge of their structure and entanglement. These methodologies have been adapted recently for solving the Navier-Stokes equations, which describe a spectrum of fluid phenomena, from the aerodynamics of vehicles to weather patterns. Within this quantum-inspired paradigm, velocity is encoded as matrix product states (MPS), effectively harnessing the analogy between interscale correlations of fluid dynamics and entanglement in quantum many-body physics. This particular tensor structure is also called quantics tensor train (QTT). By utilizing NVIDIA's cuQuantum library to perform parallel tensor computations on GPUs, our adaptation speeds up simulations by up to 12.1 times. This allows us to study the algorithm in terms of its applicability, scalability, and performance. By simulating two qualitatively different but commonly encountered 2D flow problems at high Reynolds numbers up to $1\times10^7$ using a fourth-order time stepping scheme, we find that the algorithm has a potential advantage over direct numerical simulations in the turbulent regime as the requirements for grid resolution increase drastically. In addition, we derive the scaling $χ=\mathcal{O}(\text{poly}(1/ε))$ for the maximum bond dimension $χ$ of MPS representing turbulent flow fields, with an error $ε$, based on the spectral distribution of turbulent kinetic energy. Our findings motivate further exploration of related quantum algorithms and other tensor network methods. △ Less

Submitted 25 June, 2024; originally announced June 2024.

Quantum Mini-Apps: A Framework for Developing and Benchmarking Quantum-HPC Applications

Authors: Nishant Saurabh, Pradeep Mantha, Florian J. Kiwit, Shantenu Jha, Andre Luckow

Abstract: With the increasing maturity and scale of quantum hardware and its integration into HPC systems, there is a need to develop robust techniques for developing, characterizing, and benchmarking quantum-HPC applications and middleware systems. This requires a better understanding of interaction, coupling, and common execution patterns between quantum and classical workload tasks and components. This p… ▽ More With the increasing maturity and scale of quantum hardware and its integration into HPC systems, there is a need to develop robust techniques for developing, characterizing, and benchmarking quantum-HPC applications and middleware systems. This requires a better understanding of interaction, coupling, and common execution patterns between quantum and classical workload tasks and components. This paper identifies six quantum-HPC execution motifs - recurring execution patterns characterized by distinct coupling and interaction modes. These motifs provide the basis for a suite of quantum mini-apps - simplified application prototypes that encapsulate essential characteristics of production systems. To support these developments, we introduce a mini-app framework that offers the necessary abstractions for creating and executing mini-apps across heterogeneous quantum-HPC infrastructure, making it a valuable tool for performance characterizations and middleware development. △ Less

Submitted 12 May, 2024; originally announced May 2024.

Comments: 9 pages, 4 figures

Performance Characterization of Expert Router for Scalable LLM Inference

Authors: Josef Pichlmeier, Philipp Ross, Andre Luckow

Abstract: Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs' high computational and memory demands. Specialized models optimized for specific ta… ▽ More Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs' high computational and memory demands. Specialized models optimized for specific tasks can be combined through a routing mechanism to address these challenges, creating a modular inference system. This paper introduces Expert Router, a scalable routing architecture that directs prompts to specialized expert models. We characterize multiple Expert Router configurations, including different LLama 3 models with quantized and non-quantized weights under up to 1,000 concurrent users. Our findings reveal that Expert Router introduces minimal latency overhead, with the configuration of expert models being a dominating factor in performance outcomes. High-parameter expert models deliver stable throughput and latency under moderate concurrency levels. In contrast, smaller expert models maintain competitive performance across a wider range of concurrent users compared to tensor-parallelized baseline models. This highlights the potential of Expert Router for efficient and scalable LLM deployment. △ Less

Submitted 8 October, 2024; v1 submitted 22 April, 2024; originally announced April 2024.

Towards Application-Aware Quantum Circuit Compilation

Authors: Nils Quetschlich, Florian J. Kiwit, Maximilian A. Wolf, Carlos A. Riofrio, Lukas Burgholzer, Andre Luckow, Robert Wille

Abstract: Quantum computing has made tremendous improvements in both software and hardware that have sparked interest in academia and industry to realize quantum computing applications. To this end, several steps are necessary: The underlying problem must be encoded in a quantum circuit, a suitable device must be selected to execute it, and it must be compiled accordingly. This compilation step has a signif… ▽ More Quantum computing has made tremendous improvements in both software and hardware that have sparked interest in academia and industry to realize quantum computing applications. To this end, several steps are necessary: The underlying problem must be encoded in a quantum circuit, a suitable device must be selected to execute it, and it must be compiled accordingly. This compilation step has a significant influence on the quality of the resulting solution. However, current state-of-the-art compilation tools treat the quantum circuit as a sequence of instructions without considering the actual application it realizes -- wasting a yet untapped potential to increase the solution quality. In this work, a different approach is explored that explicitly incorporates the application considered and aims to optimize its solution quality during compilation. Initial results show the benefits of this approach: For an industry-inspired application of a quantum generative model, the proposed approach outperformed Qiskit's most-optimized compilation scheme and led to better solution quality. Therefore, this work presents a first step towards application-aware compilation. △ Less

Submitted 9 June, 2024; v1 submitted 18 April, 2024; originally announced April 2024.

Comments: 8 pages, 3 figures, minor changes, to be published at IEEE International Conference on Quantum Software (QSW), 2024

Benchmarking Quantum Generative Learning: A Study on Scalability and Noise Resilience using QUARK

Authors: Florian J. Kiwit, Maximilian A. Wolf, Marwa Marso, Philipp Ross, Jeanette M. Lorenz, Carlos A. Riofrío, Andre Luckow

Abstract: Quantum computing promises a disruptive impact on machine learning algorithms, taking advantage of the exponentially large Hilbert space available. However, it is not clear how to scale quantum machine learning (QML) to industrial-level applications. This paper investigates the scalability and noise resilience of quantum generative learning applications. We consider the training performance in the… ▽ More Quantum computing promises a disruptive impact on machine learning algorithms, taking advantage of the exponentially large Hilbert space available. However, it is not clear how to scale quantum machine learning (QML) to industrial-level applications. This paper investigates the scalability and noise resilience of quantum generative learning applications. We consider the training performance in the presence of statistical noise due to finite-shot noise statistics and quantum noise due to decoherence to analyze the scalability of QML methods. We employ rigorous benchmarking techniques to track progress and identify challenges in scaling QML algorithms, and show how characterization of QML systems can be accelerated, simplified, and made reproducible when the QUARK framework is used. We show that QGANs are not as affected by the curse of dimensionality as QCBMs and to which extent QCBMs are resilient to noise. △ Less

Submitted 27 March, 2024; originally announced March 2024.

QISS: Quantum Industrial Shift Scheduling Algorithm

Authors: Anna M. Krol, Marvin Erdmann, Rajesh Mishra, Phattharaporn Singkanipa, Ewan Munro, Marcin Ziolkowski, Andre Luckow, Zaid Al-Ars

Abstract: In this paper, we show the design and implementation of a quantum algorithm for industrial shift scheduling (QISS), which uses Grover's adaptive search to tackle a common and important class of valuable, real-world combinatorial optimization problems. We give an explicit circuit construction of the Grover's oracle, incorporating the multiple constraints present in the problem, and detail the corre… ▽ More In this paper, we show the design and implementation of a quantum algorithm for industrial shift scheduling (QISS), which uses Grover's adaptive search to tackle a common and important class of valuable, real-world combinatorial optimization problems. We give an explicit circuit construction of the Grover's oracle, incorporating the multiple constraints present in the problem, and detail the corresponding logical-level resource requirements. Further, we simulate the application of QISS to specific small-scale problem instances to corroborate the performance of the algorithm, and we provide an open-source repository with our code, available on github.com/anneriet/QISS . Our work shows how complex real-world industrial optimization problems can be formulated in the context of Grover's algorithm, and paves the way towards important tasks such as physical-level resource estimation for this category of use cases. △ Less

Submitted 15 January, 2024; originally announced January 2024.

Quantum-centric Supercomputing for Materials Science: A Perspective on Challenges and Future Directions

Authors: Yuri Alexeev, Maximilian Amsler, Paul Baity, Marco Antonio Barroca, Sanzio Bassini, Torey Battelle, Daan Camps, David Casanova, Young Jai Choi, Frederic T. Chong, Charles Chung, Chris Codella, Antonio D. Corcoles, James Cruise, Alberto Di Meglio, Jonathan Dubois, Ivan Duran, Thomas Eckl, Sophia Economou, Stephan Eidenbenz, Bruce Elmegreen, Clyde Fare, Ismael Faro, Cristina Sanz Fernández, Rodrigo Neumann Barros Ferreira , et al. (102 additional authors not shown)

Abstract: Computational models are an essential tool for the design, characterization, and discovery of novel materials. Hard computational tasks in materials science stretch the limits of existing high-performance supercomputing centers, consuming much of their simulation, analysis, and data resources. Quantum computing, on the other hand, is an emerging technology with the potential to accelerate many of… ▽ More Computational models are an essential tool for the design, characterization, and discovery of novel materials. Hard computational tasks in materials science stretch the limits of existing high-performance supercomputing centers, consuming much of their simulation, analysis, and data resources. Quantum computing, on the other hand, is an emerging technology with the potential to accelerate many of the computational tasks needed for materials science. In order to do that, the quantum technology must interact with conventional high-performance computing in several ways: approximate results validation, identification of hard problems, and synergies in quantum-centric supercomputing. In this paper, we provide a perspective on how quantum-centric supercomputing can help address critical computational problems in materials science, the challenges to face in order to solve representative use cases, and new suggested directions. △ Less

Submitted 19 September, 2024; v1 submitted 14 December, 2023; originally announced December 2023.

Comments: 65 pages, 15 figures; comments welcome

Journal ref: Future Generation Computer Systems, Volume 160, November 2024, Pages 666-710

A Conceptual Architecture for a Quantum-HPC Middleware

Authors: Nishant Saurabh, Shantenu Jha, Andre Luckow

Abstract: Quantum computing promises potential for science and industry by solving certain computationally complex problems faster than classical computers. Quantum computing systems evolved from monolithic systems towards modular architectures comprising multiple quantum processing units (QPUs) coupled to classical computing nodes (HPC). With the increasing scale, middleware systems that facilitate the eff… ▽ More Quantum computing promises potential for science and industry by solving certain computationally complex problems faster than classical computers. Quantum computing systems evolved from monolithic systems towards modular architectures comprising multiple quantum processing units (QPUs) coupled to classical computing nodes (HPC). With the increasing scale, middleware systems that facilitate the efficient coupling of quantum-classical computing are becoming critical. Through an in-depth analysis of quantum applications, integration patterns and systems, we identified a gap in understanding Quantum-HPC middleware systems. We present a conceptual middleware to facilitate reasoning about quantum-classical integration and serve as the basis for a future middleware system. An essential contribution of this paper lies in leveraging well-established high-performance computing abstractions for managing workloads, tasks, and resources to integrate quantum computing into HPC systems seamlessly. △ Less

Submitted 12 August, 2023; originally announced August 2023.

Comments: 12 pages, 3 figures

ACM Class: D.m

Application-Oriented Benchmarking of Quantum Generative Learning Using QUARK

Authors: Florian J. Kiwit, Marwa Marso, Philipp Ross, Carlos A. Riofrío, Johannes Klepsch, Andre Luckow

Abstract: Benchmarking of quantum machine learning (QML) algorithms is challenging due to the complexity and variability of QML systems, e.g., regarding model ansatzes, data sets, training techniques, and hyper-parameters selection. The QUantum computing Application benchmaRK (QUARK) framework simplifies and standardizes benchmarking studies for quantum computing applications. Here, we propose several exten… ▽ More Benchmarking of quantum machine learning (QML) algorithms is challenging due to the complexity and variability of QML systems, e.g., regarding model ansatzes, data sets, training techniques, and hyper-parameters selection. The QUantum computing Application benchmaRK (QUARK) framework simplifies and standardizes benchmarking studies for quantum computing applications. Here, we propose several extensions of QUARK to include the ability to evaluate the training and deployment of quantum generative models. We describe the updated software architecture and illustrate its flexibility through several example applications: (1) We trained different quantum generative models using several circuit ansatzes, data sets, and data transformations. (2) We evaluated our models on GPU and real quantum hardware. (3) We assessed the generalization capabilities of our generative models using a broad set of metrics that capture, e.g., the novelty and validity of the generated data. △ Less

Submitted 8 August, 2023; originally announced August 2023.

Comments: 10 pages, 10 figures

MSC Class: 81-04 ACM Class: C.4

Workflows Community Summit 2022: A Roadmap Revolution

Authors: Rafael Ferreira da Silva, Rosa M. Badia, Venkat Bala, Debbie Bard, Peer-Timo Bremer, Ian Buckley, Silvina Caino-Lores, Kyle Chard, Carole Goble, Shantenu Jha, Daniel S. Katz, Daniel Laney, Manish Parashar, Frederic Suter, Nick Tyler, Thomas Uram, Ilkay Altintas, Stefan Andersson, William Arndt, Juan Aznar, Jonathan Bader, Bartosz Balis, Chris Blanton, Kelly Rosa Braghetto, Aharon Brodutch , et al. (80 additional authors not shown)

Abstract: Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and t… ▽ More Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enabling the publication of workflows and their associated products according to the FAIR principles. This document reports on discussions and findings from the 2022 international edition of the Workflows Community Summit that took place on November 29 and 30, 2022. △ Less

Submitted 31 March, 2023; originally announced April 2023.

Report number: ORNL/TM-2023/2885

A performance characterization of quantum generative models

Authors: Carlos A. Riofrío, Oliver Mitevski, Caitlin Jones, Florian Krellner, Aleksandar Vučković, Joseph Doetsch, Johannes Klepsch, Thomas Ehmer, Andre Luckow

Abstract: Quantum generative modeling is a growing area of interest for industry-relevant applications. With the field still in its infancy, there are many competing techniques. This work is an attempt to systematically compare a broad range of these techniques to guide quantum computing practitioners when deciding which models and techniques to use in their applications. We compare fundamentally different… ▽ More Quantum generative modeling is a growing area of interest for industry-relevant applications. With the field still in its infancy, there are many competing techniques. This work is an attempt to systematically compare a broad range of these techniques to guide quantum computing practitioners when deciding which models and techniques to use in their applications. We compare fundamentally different architectural ansatzes of parametric quantum circuits used for quantum generative modeling: 1. A continuous architecture, which produces continuous-valued data samples, and 2. a discrete architecture, which samples on a discrete grid. We compare the performance of different data transformations: normalization by the min-max transform or by the probability integral transform. We learn the underlying probability distribution of the data sets via two popular training methods: 1. quantum circuit Born machines (QCBM), and 2. quantum generative adversarial networks (QGAN). We study their performance and trade-offs as the number of model parameters increases, with the baseline of similarly trained classical neural networks. The study is performed on six low-dimensional synthetic and two real financial data sets. Our two key findings are that: 1. For all data sets, our quantum models require similar or fewer parameters than their classical counterparts. In the extreme case, the quantum models require two of orders of magnitude less parameters. 2. We empirically find that a variant of the discrete architecture, which learns the copula of the probability distribution, outperforms all other methods. △ Less

Submitted 26 March, 2024; v1 submitted 23 January, 2023; originally announced January 2023.

Comments: Revised version: Small corrections to figures, additional references, and link to open-source code associated with the project. This version is in line with the version accepted for publication

Report number: Article No.: 12, Pages 1 - 34

Journal ref: ACM Transactions on Quantum Computing, Volume 5, Issue 2 (2024)

Quantum Computing Techniques for Multi-Knapsack Problems

Authors: Abhishek Awasthi, Francesco Bär, Joseph Doetsch, Hans Ehm, Marvin Erdmann, Maximilian Hess, Johannes Klepsch, Peter A. Limacher, Andre Luckow, Christoph Niedermeier, Lilly Palackal, Ruben Pfeiffer, Philipp Ross, Hila Safi, Janik Schönmeier-Kromer, Oliver von Sicard, Yannick Wenger, Karen Wintersperger, Sheir Yarkoni

Abstract: Optimization problems are ubiquitous in various industrial settings, and multi-knapsack optimization is one recurrent task faced daily by several industries. The advent of quantum computing has opened a new paradigm for computationally intensive tasks, with promises of delivering better and faster solutions for specific classes of problems. This work presents a comprehensive study of quantum compu… ▽ More Optimization problems are ubiquitous in various industrial settings, and multi-knapsack optimization is one recurrent task faced daily by several industries. The advent of quantum computing has opened a new paradigm for computationally intensive tasks, with promises of delivering better and faster solutions for specific classes of problems. This work presents a comprehensive study of quantum computing approaches for multi-knapsack problems, by investigating some of the most prominent and state-of-the-art quantum algorithms using different quantum software and hardware tools. The performance of the quantum approaches is compared for varying hyperparameters. We consider several gate-based quantum algorithms, such as QAOA and VQE, as well as quantum annealing, and present an exhaustive study of the solutions and the estimation of runtimes. Additionally, we analyze the impact of warm-starting QAOA to understand the reasons for the better performance of this approach. We discuss the implications of our results in view of utilizing quantum optimization for industrial applications in the future. In addition to the high demand for better quantum hardware, our results also emphasize the necessity of more and better quantum optimization algorithms, especially for multi-knapsack problems. △ Less

Submitted 28 September, 2023; v1 submitted 13 January, 2023; originally announced January 2023.

Comments: 20 pages

Journal ref: Arai, K. (eds) Intelligent Computing. SAI 2023. Lecture Notes in Networks and Systems, vol 739

Exploring privacy-enhancing technologies in the automotive value chain

Authors: Gonzalo Munilla Garrido, Kaja Schmidt, Christopher Harth-Kitzerow, Johannes Klepsch, Andre Luckow, Florian Matthes

Abstract: Privacy-enhancing technologies (PETs) are becoming increasingly crucial for addressing customer needs, security, privacy (e.g., enhancing anonymity and confidentiality), and regulatory requirements. However, applying PETs in organizations requires a precise understanding of use cases, technologies, and limitations. This paper investigates several industrial use cases, their characteristics, and th… ▽ More Privacy-enhancing technologies (PETs) are becoming increasingly crucial for addressing customer needs, security, privacy (e.g., enhancing anonymity and confidentiality), and regulatory requirements. However, applying PETs in organizations requires a precise understanding of use cases, technologies, and limitations. This paper investigates several industrial use cases, their characteristics, and the potential applicability of PETs to these. We conduct expert interviews to identify and classify uses cases, a gray literature review of relevant open-source PET tools, and discuss how the use case characteristics can be addressed using PETs' capabilities. While we focus mainly on automotive use cases, the results also apply to other use case domains. △ Less

Submitted 12 September, 2022; originally announced September 2022.

Journal ref: 2021 IEEE International Conference on Big Data (Big Data)

Optimization of Robot Trajectory Planning with Nature-Inspired and Hybrid Quantum Algorithms

Authors: Martin J. A. Schuetz, J. Kyle Brubaker, Henry Montagu, Yannick van Dijk, Johannes Klepsch, Philipp Ross, Andre Luckow, Mauricio G. C. Resende, Helmut G. Katzgraber

Abstract: We solve robot trajectory planning problems at industry-relevant scales. Our end-to-end solution integrates highly versatile random-key algorithms with model stacking and ensemble techniques, as well as path relinking for solution refinement. The core optimization module consists of a biased random-key genetic algorithm. Through a distinct separation of problem-independent and problem-dependent mo… ▽ More We solve robot trajectory planning problems at industry-relevant scales. Our end-to-end solution integrates highly versatile random-key algorithms with model stacking and ensemble techniques, as well as path relinking for solution refinement. The core optimization module consists of a biased random-key genetic algorithm. Through a distinct separation of problem-independent and problem-dependent modules, we achieve an efficient problem representation, with a native encoding of constraints. We show that generalizations to alternative algorithmic paradigms such as simulated annealing are straightforward. We provide numerical benchmark results for industry-scale data sets. Our approach is found to consistently outperform greedy baseline results. To assess the capabilities of today's quantum hardware, we complement the classical approach with results obtained on quantum annealing hardware, using qbsolv on Amazon Braket. Finally, we show how the latter can be integrated into our larger pipeline, providing a quantum-ready hybrid solution to the problem. △ Less

Submitted 7 June, 2022; originally announced June 2022.

Comments: 17 pages, 6 figures

Journal ref: Phys. Rev. Applied 18, 054045 (2022)

CRGC -- A Practical Framework for Constructing Reusable Garbled Circuits

Authors: Christopher Harth-Kitzerow, Georg Carle, Fan Fei, Andre Luckow, Johannes Klepsch

Abstract: In this work, we introduce two schemes to construct reusable garbled circuits (RGCs) in the semi-honest setting. Our completely reusable garbled circuit (CRGC) scheme allows the generator (party A) to construct and send an obfuscated boolean circuit along with an encoded input to the evaluator (party B). In contrast to Yao's Garbled Circuit protocol, B can securely evaluate the same CRGC with an a… ▽ More In this work, we introduce two schemes to construct reusable garbled circuits (RGCs) in the semi-honest setting. Our completely reusable garbled circuit (CRGC) scheme allows the generator (party A) to construct and send an obfuscated boolean circuit along with an encoded input to the evaluator (party B). In contrast to Yao's Garbled Circuit protocol, B can securely evaluate the same CRGC with an arbitrary number of inputs. As a tradeoff, CRGCs predictably leak some input bits of A to B. We also propose a partially reusable garbled circuit (PRGC) scheme that divides a circuit into reusable and non-reusable sections. PRGCs do not leak input bits of A. We benchmark our CRGC implementation against the state-of-the-art garbled circuit libraries EMP SH2PC and TinyGarble2. Using our framework, evaluating a CRGC is up to twenty times faster, albeit with weaker privacy guarantees, than evaluating an equivalent garbled circuit constructed by the two existing libraries. Our open-source library can convert any C++ function to a CRGC at approx. 80 million gates per second and repeatedly evaluate a CRGC at approx. 350 million gates per second. Additionally, a compressed CRGC is approx. 75% smaller in file size than the unobfuscated boolean circuit. △ Less

Submitted 6 May, 2022; v1 submitted 23 March, 2022; originally announced March 2022.

Comments: 13 pages, 7 figures

QUARK: A Framework for Quantum Computing Application Benchmarking

Authors: Jernej Rudi Finžgar, Philipp Ross, Leonhard Hölscher, Johannes Klepsch, Andre Luckow

Abstract: Quantum computing (QC) is anticipated to provide a speedup over classical HPC approaches for specific problems in optimization, simulation, and machine learning. With the advances in quantum computing toward practical applications, the need to analyze and compare different quantum solutions increases. While different low-level benchmarks for QC exist, these benchmarks do not provide sufficient ins… ▽ More Quantum computing (QC) is anticipated to provide a speedup over classical HPC approaches for specific problems in optimization, simulation, and machine learning. With the advances in quantum computing toward practical applications, the need to analyze and compare different quantum solutions increases. While different low-level benchmarks for QC exist, these benchmarks do not provide sufficient insights into real-world application-level performance. We propose an application-centric benchmark method and the QUantum computing Application benchmaRK (QUARK) framework to foster the investigation and creation of application benchmarks for QC. This paper establishes three significant contributions: (1) it makes a case for application-level benchmarks and provides an in-depth "pen and paper" benchmark formulation of two reference problems: robot path and vehicle option optimization from the industrial domain; (2) it proposes the open-source QUARK framework for designing, implementing, executing, and analyzing benchmarks; (3) it provides multiple reference implementations for these two reference problems based on different known, and where needed, extended, classical and quantum algorithmic approaches and analyzes their performance on different types of infrastructures. △ Less

Submitted 5 August, 2022; v1 submitted 7 February, 2022; originally announced February 2022.

Comments: Improved version, as submitted to IEEE QCE22 conference

Journal ref: IEEE QCE2022

Revealing the Landscape of Privacy-Enhancing Technologies in the Context of Data Markets for the IoT: A Systematic Literature Review

Authors: Gonzalo Munilla Garrido, Johannes Sedlmeir, Ömer Uludağ, Ilias Soto Alaoui, Andre Luckow, Florian Matthes

Abstract: IoT data markets in public and private institutions have become increasingly relevant in recent years because of their potential to improve data availability and unlock new business models. However, exchanging data in markets bears considerable challenges related to disclosing sensitive information. Despite considerable research focused on different aspects of privacy-enhancing data markets for th… ▽ More IoT data markets in public and private institutions have become increasingly relevant in recent years because of their potential to improve data availability and unlock new business models. However, exchanging data in markets bears considerable challenges related to disclosing sensitive information. Despite considerable research focused on different aspects of privacy-enhancing data markets for the IoT, none of the solutions proposed so far seems to find a practical adoption. Thus, this study aims to organize the state-of-the-art solutions, analyze and scope the technologies that have been suggested in this context, and structure the remaining challenges to determine areas where future research is required. To accomplish this goal, we conducted a systematic literature review on privacy enhancement in data markets for the IoT, covering 50 publications dated up to July 2020, and provided updates with 24 publications dated up to May 2022. Our results indicate that most research in this area has emerged only recently, and no IoT data market architecture has established itself as canonical. Existing solutions frequently lack the required combination of anonymization and secure computation technologies. Furthermore, there is no consensus on the appropriate use of blockchain technology for IoT data markets and a low degree of leveraging existing libraries or reusing generic data market architectures. We also identified significant challenges remaining, such as the copy problem and the recursive enforcement problem that-while solutions have been suggested to some extent-are often not sufficiently addressed in proposed designs. We conclude that privacy-enhancing technologies need further improvements to positively impact data markets so that, ultimately, the value of data is preserved through data scarcity and users' privacy and businesses-critical information are protected. △ Less

Submitted 12 July, 2022; v1 submitted 25 July, 2021; originally announced July 2021.

Comments: 49 pages, 17 figures, 11 tables

Pilot-Edge: Distributed Resource Management Along the Edge-to-Cloud Continuum

Authors: Andre Luckow, Kartik Rattan, Shantenu Jha

Abstract: Many science and industry IoT applications necessitate data processing across the edge-to-cloud continuum to meet performance, security, cost, and privacy requirements. However, diverse abstractions and infrastructures for managing resources and tasks across the edge-to-cloud scenario are required. We propose Pilot-Edge as a common abstraction for resource management across the edge-to-cloud conti… ▽ More Many science and industry IoT applications necessitate data processing across the edge-to-cloud continuum to meet performance, security, cost, and privacy requirements. However, diverse abstractions and infrastructures for managing resources and tasks across the edge-to-cloud scenario are required. We propose Pilot-Edge as a common abstraction for resource management across the edge-to-cloud continuum. Pilot-Edge is based on the pilot abstraction, which decouples resource and workload management, and provides a Function-as-a-Service (FaaS) interface for application-level tasks. The abstraction allows applications to encapsulate common functions in high-level tasks that can then be configured and deployed across the continuum. We characterize Pilot-Edge on geographically distributed infrastructures using machine learning workloads (e.g., k-means and auto-encoders). Our experiments demonstrate how Pilot-Edge manages distributed resources and allows applications to evaluate task placement based on multiple factors (e.g., model complexities, throughput, and latency). △ Less

Submitted 7 April, 2021; originally announced April 2021.

Comments: 5 pages, 3 figures

ACM Class: C.4; C.2.4

Exploring Task Placement for Edge-to-Cloud Applications using Emulation

Authors: Andre Luckow, Kartik Rattan, Shantenu Jha

Abstract: A vast and growing number of IoT applications connect physical devices, such as scientific instruments, technical equipment, machines, and cameras, across heterogenous infrastructure from the edge to the cloud to provide responsive, intelligent services while complying with privacy and security requirements. However, the integration of heterogeneous IoT, edge, and cloud technologies and the design… ▽ More A vast and growing number of IoT applications connect physical devices, such as scientific instruments, technical equipment, machines, and cameras, across heterogenous infrastructure from the edge to the cloud to provide responsive, intelligent services while complying with privacy and security requirements. However, the integration of heterogeneous IoT, edge, and cloud technologies and the design of end-to-end applications that seamlessly work across multiple layers and types of infrastructures is challenging. A significant issue is resource management and the need to ensure that the right type and scale of resources is allocated on every layer to fulfill the application's processing needs. As edge and cloud layers are increasingly tightly integrated, imbalanced resource allocations and sub-optimally placed tasks can quickly deteriorate the overall system performance. This paper proposes an emulation approach for the investigation of task placements across the edge-to-cloud continuum. We demonstrate that emulation can address the complexity and many degrees-of-freedom of the problem, allowing us to investigate essential deployment patterns and trade-offs. We evaluate our approach using a machine learning-based workload, demonstrating the validity by comparing emulation and real-world experiments. Further, we show that the right task placement strategy has a significant impact on performance -- in our experiments, between 5% and 65% depending on the scenario. △ Less

Submitted 7 April, 2021; originally announced April 2021.

Comments: 5 pages, 2 figures

ACM Class: C.4

Quantum Computing: Towards Industry Reference Problems

Authors: Andre Luckow, Johannes Klepsch, Josef Pichlmeier

Abstract: The complexity is increasing rapidly in many areas of the automotive industry. The design of an automobile involves many different engineering disciplines, e. g., mechanical, electrical, and software engineering. The software of a vehicle comprises millions of lines of code. Further, the manufacturing, logistics, distribution, and sales of a vehicle are highly complex. There is an immense need for… ▽ More The complexity is increasing rapidly in many areas of the automotive industry. The design of an automobile involves many different engineering disciplines, e. g., mechanical, electrical, and software engineering. The software of a vehicle comprises millions of lines of code. Further, the manufacturing, logistics, distribution, and sales of a vehicle are highly complex. There is an immense need for solving simulation problems, e. g., in battery chemistry, an essential enabler for technological advancements for electric vehicles. In all these domains, myriads of optimization, simulation, and machine learning problems arise. Quantum computing-based approaches promise to overcome some of the inherent scalability limitations of classical approaches. This article investigates quantum computing applications across the automotive value chain and identifies several high-value problems that will benefit from quantum-enhanced solutions. △ Less

Submitted 12 March, 2021; originally announced March 2021.

Comments: This is a pre-print of an article published in DIGITALE WELT Volume 5, issue 2. The final authenticated version is available online at: https://doi.org/10.1007/s42354-021-0335-7

Journal ref: Digitale Welt 5, 38-45 (2021)

An In-Depth Investigation of the Performance Characteristics of Hyperledger Fabric

Authors: Tobias Guggenberger, Johannes Sedlmeir, Gilbert Fridgen, André Luckow

Abstract: Private permissioned blockchains are deployed in ever greater numbers to facilitate cross-organizational processes in various industries, particularly in supply chain management. One popular example of this trend is Hyperledger Fabric. Compared to public permissionless blockchains, it promises improved performance and provides certain features that address key requirements of enterprises. However,… ▽ More Private permissioned blockchains are deployed in ever greater numbers to facilitate cross-organizational processes in various industries, particularly in supply chain management. One popular example of this trend is Hyperledger Fabric. Compared to public permissionless blockchains, it promises improved performance and provides certain features that address key requirements of enterprises. However, also permissioned blockchains are still not as scalable as centralized systems, and due to the scarcity of theoretical results and empirical data, their real-world performance cannot be predicted with the necessary precision. We intend to address this issue by conducting an in-depth performance analysis of Hyperledger Fabric. The paper presents a detailed compilation of various performance characteristics using an enhanced version of the Distributed Ledger Performance Scan (DLPS). Researchers and practitioners alike can use the various performance properties identified and discussed as guidelines to better configure and implement their Hyperledger Fabric network. Likewise, they are encouraged to use the DLPS framework to conduct their measurements. △ Less

Submitted 3 January, 2023; v1 submitted 15 February, 2021; originally announced February 2021.

ACM Class: H.4; J.1

Journal ref: Computers & Industrial Engineering (2022), Volume 173, 108716

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Authors: Andre Luckow, Shantenu Jha

Abstract: Developing software for scientific applications that require the integration of diverse types of computing, instruments, and data present challenges that are distinct from commercial software. These applications require scale, and the need to integrate various programming and computational models with evolving and heterogeneous infrastructure. Pervasive and effective abstractions for distributed i… ▽ More Developing software for scientific applications that require the integration of diverse types of computing, instruments, and data present challenges that are distinct from commercial software. These applications require scale, and the need to integrate various programming and computational models with evolving and heterogeneous infrastructure. Pervasive and effective abstractions for distributed infrastructures are thus critical; however, the process of developing abstractions for scientific applications and infrastructures is not well understood. While theory-based approaches for system development are suited for well-defined, closed environments, they have severe limitations for designing abstractions for scientific systems and applications. The design science research (DSR) method provides the basis for designing practical systems that can handle real-world complexities at all levels. In contrast to theory-centric approaches, DSR emphasizes both practical relevance and knowledge creation by building and rigorously evaluating all artifacts. We show how DSR provides a well-defined framework for developing abstractions and middleware systems for distributed systems. Specifically, we address the critical problem of distributed resource management on heterogeneous infrastructure over a dynamic range of scales, a challenge that currently limits many scientific applications. We use the pilot-abstraction, a widely used resource management abstraction for high-performance, high throughput, big data, and streaming applications, as a case study for evaluating the DSR activities. For this purpose, we analyze the research process and artifacts produced during the design and evaluation of the pilot-abstraction. We find DSR provides a concise framework for iteratively designing and evaluating systems. Finally, we capture our experiences and formulate different lessons learned. △ Less

Submitted 25 March, 2020; v1 submitted 20 February, 2020; originally announced February 2020.

Comments: 10 pages, 5 figures

Performance Characterization and Modeling of Serverless and HPC Streaming Applications

Authors: Andre Luckow, Shantenu Jha

Abstract: Experiment-in-the-Loop Computing (EILC) requires support for numerous types of processing and the management of heterogeneous infrastructure over a dynamic range of scales: from the edge to the cloud and HPC, and intermediate resources. Serverless is an emerging service that combines high-level middleware services, such as distributed execution engines for managing tasks, with low-level infrastruc… ▽ More Experiment-in-the-Loop Computing (EILC) requires support for numerous types of processing and the management of heterogeneous infrastructure over a dynamic range of scales: from the edge to the cloud and HPC, and intermediate resources. Serverless is an emerging service that combines high-level middleware services, such as distributed execution engines for managing tasks, with low-level infrastructure. It offers the potential of usability and scalability, but adds to the complexity of managing heterogeneous and dynamic resources. In response, we extend Pilot-Streaming to support serverless platforms. Pilot-Streaming provides a unified abstraction for resource management for HPC, cloud, and serverless, and allocates resource containers independent of the application workload removing the need to write resource-specific code. Understanding of the performance and scaling characteristics of streaming applications and infrastructure presents another challenge for EILC. StreamInsight provides insight into the performance of streaming applications and infrastructure, their selection, configuration and scaling behavior. Underlying StreamInsight is the universal scalability law, which permits the accurate quantification of scalability properties of streaming applications. Using experiments on HPC and AWS Lambda, we demonstrate that StreamInsight provides an accurate model for a variety of application characteristics, e.g., machine learning model sizes and resource configurations. △ Less

Submitted 13 September, 2019; originally announced September 2019.

Pilot-Streaming: A Stream Processing Framework for High-Performance Computing

Authors: Andre Luckow, George Chantzialexiou, Shantenu Jha

Abstract: An increasing number of scientific applications rely on stream processing for generating timely insights from data feeds of scientific instruments, simulations, and Internet-of-Thing (IoT) sensors. The development of streaming applications is a complex task and requires the integration of heterogeneous, distributed infrastructure, frameworks, middleware and application components. Different applic… ▽ More An increasing number of scientific applications rely on stream processing for generating timely insights from data feeds of scientific instruments, simulations, and Internet-of-Thing (IoT) sensors. The development of streaming applications is a complex task and requires the integration of heterogeneous, distributed infrastructure, frameworks, middleware and application components. Different application components are often written in different languages using different abstractions and frameworks. Often, additional components, such as a message broker (e.g. Kafka), are required to decouple data production and consumptions and avoiding issues, such as back-pressure. Streaming applications may be extremely dynamic due to factors, such as variable data rates caused by the data source, adaptive sampling techniques or network congestions, variable processing loads caused by usage of different machine learning algorithms. As a result application-level resource management that can respond to changes in one of these factors is critical. We propose Pilot-Streaming, a framework for supporting streaming frameworks, applications and their resource management needs on HPC infrastructure. Pilot-Streaming is based on the Pilot-Job concept and enables developers to manage distributed computing and data resources for complex streaming applications. It enables applications to dynamically respond to resource requirements by adding/removing resources at runtime. This capability is critical for balancing complex streaming pipelines. To address the complexity in developing and characterization of streaming applications, we present the Streaming Mini- App framework, which supports different plug-able algorithms for data generation and processing, e.g., for reconstructing light source images using different techniques. We utilize the Mini-App framework to conduct an evaluation of Pilot-Streaming. △ Less

Submitted 11 November, 2018; v1 submitted 25 January, 2018; originally announced January 2018.

Comments: 12 pages

Task-parallel Analysis of Molecular Dynamics Trajectories

Authors: Ioannis Paraskevakos, Andre Luckow, Mahzad Khoshlessan, George Chantzialexiou, Thomas E. Cheatham, Oliver Beckstein, Geoffrey C. Fox, Shantenu Jha

Abstract: Different parallel frameworks for implementing data analysis applications have been proposed by the HPC and Big Data communities. In this paper, we investigate three task-parallel frameworks: Spark, Dask and RADICAL-Pilot with respect to their ability to support data analytics on HPC resources and compare them with MPI. We investigate the data analysis requirements of Molecular Dynamics (MD) simul… ▽ More Different parallel frameworks for implementing data analysis applications have been proposed by the HPC and Big Data communities. In this paper, we investigate three task-parallel frameworks: Spark, Dask and RADICAL-Pilot with respect to their ability to support data analytics on HPC resources and compare them with MPI. We investigate the data analysis requirements of Molecular Dynamics (MD) simulations which are significant consumers of supercomputing cycles, producing immense amounts of data. A typical large-scale MD simulation of a physical system of O(100k) atoms over μsecs can produce from O(10) GB to O(1000) GBs of data. We propose and evaluate different approaches for parallelization of a representative set of MD trajectory analysis algorithms, in particular the computation of path similarity and leaflet identification. We evaluate Spark, Dask and RADICAL-Pilot with respect to their abstractions and runtime engine capabilities to support these algorithms. We provide a conceptual basis for comparing and understanding different frameworks that enable users to select the optimal system for each application. We also provide a quantitative performance analysis of the different algorithms across the three frameworks. △ Less

Submitted 10 June, 2018; v1 submitted 23 January, 2018; originally announced January 2018.

Deep Learning in the Automotive Industry: Applications and Tools

Authors: Andre Luckow, Matthew Cook, Nathan Ashcraft, Edwin Weill, Emil Djerekarov, Bennie Vorster

Abstract: Deep Learning refers to a set of machine learning techniques that utilize neural networks with many hidden layers for tasks, such as image classification, speech recognition, language understanding. Deep learning has been proven to be very effective in these domains and is pervasively used by many Internet services. In this paper, we describe different automotive uses cases for deep learning in pa… ▽ More Deep Learning refers to a set of machine learning techniques that utilize neural networks with many hidden layers for tasks, such as image classification, speech recognition, language understanding. Deep learning has been proven to be very effective in these domains and is pervasively used by many Internet services. In this paper, we describe different automotive uses cases for deep learning in particular in the domain of computer vision. We surveys the current state-of-the-art in libraries, tools and infrastructures (e.\,g.\ GPUs and clouds) for implementing, training and deploying deep neural networks. We particularly focus on convolutional neural networks and computer vision use cases, such as the visual inspection process in manufacturing plants and the analysis of social media data. To train neural networks, curated and labeled datasets are essential. In particular, both the availability and scope of such datasets is typically very limited. A main contribution of this paper is the creation of an automotive dataset, that allows us to learn and automatically recognize different vehicle properties. We describe an end-to-end deep learning application utilizing a mobile app for data collection and process support, and an Amazon-based cloud backend for storage and training. For training we evaluate the use of cloud and on-premises infrastructures (including multiple GPUs) in conjunction with different neural network architectures and frameworks. We assess both the training times as well as the accuracy of the classifier. Finally, we demonstrate the effectiveness of the trained classifier in a real world setting during manufacturing process. △ Less

Submitted 30 April, 2017; originally announced May 2017.

Comments: 10 pages

Algebraic multigrid support vector machines

Authors: Ehsan Sadrfaridpour, Sandeep Jeereddy, Ken Kennedy, Andre Luckow, Talayeh Razzaghi, Ilya Safro

Abstract: The support vector machine is a flexible optimization-based technique widely used for classification problems. In practice, its training part becomes computationally expensive on large-scale data sets because of such reasons as the complexity and number of iterations in parameter fitting methods, underlying optimization solvers, and nonlinearity of kernels. We introduce a fast multilevel framework… ▽ More The support vector machine is a flexible optimization-based technique widely used for classification problems. In practice, its training part becomes computationally expensive on large-scale data sets because of such reasons as the complexity and number of iterations in parameter fitting methods, underlying optimization solvers, and nonlinearity of kernels. We introduce a fast multilevel framework for solving support vector machine models that is inspired by the algebraic multigrid. Significant improvement in the running has been achieved without any loss in the quality. The proposed technique is highly beneficial on imbalanced sets. We demonstrate computational results on publicly available and industrial data sets. △ Less

Submitted 23 November, 2016; v1 submitted 16 November, 2016; originally announced November 2016.

Introducing Distributed Dynamic Data-intensive (D3) Science: Understanding Applications and Infrastructure

Authors: Shantenu Jha, Daniel S. Katz, Andre Luckow, Omer Rana, Yogesh Simmhan, Neil Chue Hong

Abstract: A common feature across many science and engineering applications is the amount and diversity of data and computation that must be integrated to yield insights. Data sets are growing larger and becoming distributed; and their location, availability and properties are often time-dependent. Collectively, these characteristics give rise to dynamic distributed data-intensive applications. While "stati… ▽ More A common feature across many science and engineering applications is the amount and diversity of data and computation that must be integrated to yield insights. Data sets are growing larger and becoming distributed; and their location, availability and properties are often time-dependent. Collectively, these characteristics give rise to dynamic distributed data-intensive applications. While "static" data applications have received significant attention, the characteristics, requirements, and software systems for the analysis of large volumes of dynamic, distributed data, and data-intensive applications have received relatively less attention. This paper surveys several representative dynamic distributed data-intensive application scenarios, provides a common conceptual framework to understand them, and examines the infrastructure used in support of applications. △ Less

Submitted 12 September, 2016; originally announced September 2016.

Comments: 38 pages, 2 figures

Hadoop on HPC: Integrating Hadoop and Pilot-based Dynamic Resource Management

Authors: Andre Luckow, Ioannis Paraskevakos, George Chantzialexiou, Shantenu Jha

Abstract: High-performance computing platforms such as supercomputers have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the limitations of HPC platforms. There exist a… ▽ More High-performance computing platforms such as supercomputers have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the limitations of HPC platforms. There exist a class of scientific applications however, that need the collective capabilities of traditional high-performance computing environments and the Apache Hadoop ecosystem. For example, the scientific domains of bio-molecular dynamics, genomics and network science need to couple traditional computing with Hadoop/Spark based analysis. We investigate the critical question of how to present the capabilities of both computing environments to such scientific applications. Whereas this questions needs answers at multiple levels, we focus on the design of resource management middleware that might support the needs of both. We propose extensions to the Pilot-Abstraction to provide a unifying resource management layer. This is an important step that allows applications to integrate HPC stages (e.g. simulations) to data analytics. Many supercomputing centers have started to officially support Hadoop environments, either in a dedicated environment or in hybrid deployments using tools such as myHadoop. This typically involves many intrinsic, environment-specific details that need to be mastered, and often swamp conceptual issues like: How best to couple HPC and Hadoop application stages? How to explore runtime trade-offs (data localities vs. data movement)? This paper provides both conceptual understanding and practical solutions to the integrated use of HPC and Hadoop environments. △ Less

Submitted 31 January, 2016; originally announced February 2016.

Pilot-Abstraction: A Valid Abstraction for Data-Intensive Applications on HPC, Hadoop and Cloud Infrastructures?

Authors: Andre Luckow, Pradeep Mantha, Shantenu Jha

Abstract: HPC environments have traditionally been designed to meet the compute demand of scientific applications and data has only been a second order concern. With science moving toward data-driven discoveries relying more on correlations in data to form scientific hypotheses, the limitations of HPC approaches become apparent: Architectural paradigms such as the separation of storage and compute are not o… ▽ More HPC environments have traditionally been designed to meet the compute demand of scientific applications and data has only been a second order concern. With science moving toward data-driven discoveries relying more on correlations in data to form scientific hypotheses, the limitations of HPC approaches become apparent: Architectural paradigms such as the separation of storage and compute are not optimal for I/O intensive workloads (e.g. for data preparation, transformation and SQL). While there are many powerful computational and analytical libraries available on HPC (e.g. for scalable linear algebra), they generally lack the usability and variety of analytical libraries found in other environments (e.g. the Apache Hadoop ecosystem). Further, there is a lack of abstractions that unify access to increasingly heterogeneous infrastructure (HPC, Hadoop, clouds) and allow reasoning about performance trade-offs in this complex environment. At the same time, the Hadoop ecosystem is evolving rapidly and has established itself as de-facto standard for data-intensive workloads in industry and is increasingly used to tackle scientific problems. In this paper, we explore paths to interoperability between Hadoop and HPC, examine the differences and challenges, such as the different architectural paradigms and abstractions, and investigate ways to address them. We propose the extension of the Pilot-Abstraction to Hadoop to serve as interoperability layer for allocating and managing resources across different infrastructures. Further, in-memory capabilities have been deployed to enhance the performance of large-scale data analytics (e.g. iterative algorithms) for which the ability to re-use data across iterations is critical. As memory naturally fits in with the Pilot concept of retaining resources for a set of tasks, we propose the extension of the Pilot-Abstraction to in-memory resources. △ Less

Submitted 20 January, 2015; originally announced January 2015.

Comments: Submitted to HPDC 2015, 12 pages, 9 figures

ACM Class: C.1.4; C.2.4; D.1.3; D.2.12

A Tale of Two Data-Intensive Paradigms: Applications, Abstractions, and Architectures

Authors: Shantenu Jha, Judy Qiu, Andre Luckow, Pradeep Mantha, Geoffrey C. Fox

Abstract: Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, co-placement and scheduling of data with compute resources, and storing and transferring large volumes of data. We analyze the ecosystems of the two prominent paradigms for data-intensive applications, hereafter referred to as the high-perform… ▽ More Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, co-placement and scheduling of data with compute resources, and storing and transferring large volumes of data. We analyze the ecosystems of the two prominent paradigms for data-intensive applications, hereafter referred to as the high-performance computing and the Apache-Hadoop paradigm. We propose a basis, common terminology and functional factors upon which to analyze the two approaches of both paradigms. We discuss the concept of "Big Data Ogres" and their facets as means of understanding and characterizing the most common application workloads found across the two paradigms. We then discuss the salient features of the two paradigms, and compare and contrast the two approaches. Specifically, we examine common implementation/approaches of these paradigms, shed light upon the reasons for their current "architecture" and discuss some typical workloads that utilize them. In spite of the significant software distinctions, we believe there is architectural similarity. We discuss the potential integration of different implementations, across the different levels and components. Our comparison progresses from a fully qualitative examination of the two paradigms, to a semi-quantitative methodology. We use a simple and broadly used Ogre (K-means clustering), characterize its performance on a range of representative platforms, covering several implementations from both paradigms. Our experiments provide an insight into the relative strengths of the two paradigms. We propose that the set of Ogres will serve as a benchmark to evaluate the two paradigms along different dimensions. △ Less

Submitted 22 June, 2014; v1 submitted 6 March, 2014; originally announced March 2014.

Comments: 8 pages, 2 figures

Pilot-Data: An Abstraction for Distributed Data

Authors: Andre Luckow, Mark Santcroos, Ashley Zebrowski, Shantenu Jha

Abstract: Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, controlling co-placement and scheduling of data with compute resources, and storing, transferring, and managing large volumes of data. Although there exist multiple approaches to addressing each of these challenges, an integrative approach is… ▽ More Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, controlling co-placement and scheduling of data with compute resources, and storing, transferring, and managing large volumes of data. Although there exist multiple approaches to addressing each of these challenges, an integrative approach is missing; furthermore, extending existing functionality or enabling interoperable capabilities remains difficult at best. We propose the concept of Pilot-Data to address the fundamental challenges of co-placement and scheduling of data and compute in heterogeneous and distributed environments with interoperability and extensibility as first-order concerns. Pilot-Data is an extension of the Pilot-Job abstraction for supporting the management of data in conjunction with compute tasks. Pilot-Data separates logical data units from physical storage, thereby providing the basis for efficient compute/data placement and scheduling. In this paper, we discuss the design and implementation of the Pilot-Data prototype, demonstrate its use by data-intensive applications on multiple production distributed cyberinfrastructure and illustrate the advantages arising from flexible execution modes enabled by Pilot-Data. Our experiments utilize an implementation of Pilot-Data in conjunction with a scalable Pilot-Job (BigJob) to establish the application performance that can be enabled by the use of Pilot-Data. We demonstrate how the concept of Pilot-Data also provides the basis upon which to build tools and support capabilities like affinity which in turn can be used for advanced data-compute co-placement and scheduling. △ Less

Submitted 18 November, 2013; v1 submitted 26 January, 2013; originally announced January 2013.

ACM Class: C.2.4

P*: A Model of Pilot-Abstractions

Authors: Andre Luckow, Mark Santcroos, Ole Weidner, Andre Merzky, Pradeep Mantha, Shantenu Jha

Abstract: Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jo… ▽ More Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jobs which can be used to define, compare and contrast different implementations. Often Pilot-Job implementations are strongly coupled to the distributed cyber-infrastructure they were originally designed for. These factors present a barrier to extensibility and interoperability. This pa- per is an attempt to (i) provide a minimal but complete model (P*) of Pilot-Jobs, (ii) establish the generality of the P* Model by mapping various existing and well known Pilot-Job frameworks such as Condor and DIANE to P*, (iii) derive an interoperable and extensible API for the P* Model (Pilot-API), (iv) validate the implementation of the Pilot-API by concurrently using multiple distinct Pilot-Job frameworks on distinct production distributed cyberinfrastructures, and (v) apply the P* Model to Pilot-Data. △ Less

Submitted 27 July, 2012; originally announced July 2012.

Comments: 10 pages