-
Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures
Authors:
Ozgur O. Kilic,
David K. Park,
Yihui Ren,
Tatiana Korchuganova,
Sairam Sri Vatsavai,
Joseph Boudreau,
Tasnuva Chowdhury,
Shengyu Feng,
Raees Khan,
Jaehyung Kim,
Scott Klasky,
Tadashi Maeno,
Paul Nilsson,
Verena Ingrid Martinez Outschoorn,
Norbert Podhorszki,
Frédéric Suter,
Wei Yang,
Yiming Yang,
Shinjae Yoo,
Alexei Klimentov,
Adolfy Hoisie
Abstract:
Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived…
▽ More
Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load-derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.
△ Less
Submitted 24 June, 2025;
originally announced June 2025.
-
Alternative Mixed Integer Linear Programming Optimization for Joint Job Scheduling and Data Allocation in Grid Computing
Authors:
Shengyu Feng,
Jaehyung Kim,
Yiming Yang,
Joseph Boudreau,
Tasnuva Chowdhury,
Adolfy Hoisie,
Raees Khan,
Ozgur O. Kilic,
Scott Klasky,
Tatiana Korchuganova,
Paul Nilsson,
Verena Ingrid Martinez Outschoorn,
David K. Park,
Norbert Podhorszki,
Yihui Ren,
Frederic Suter,
Sairam Sri Vatsavai,
Wei Yang,
Shinjae Yoo,
Tadashi Maeno,
Alexei Klimentov
Abstract:
This paper presents a novel approach to the joint optimization of job scheduling and data allocation in grid computing environments. We formulate this joint optimization problem as a mixed integer quadratically constrained program. To tackle the nonlinearity in the constraint, we alternatively fix a subset of decision variables and optimize the remaining ones via Mixed Integer Linear Programming (…
▽ More
This paper presents a novel approach to the joint optimization of job scheduling and data allocation in grid computing environments. We formulate this joint optimization problem as a mixed integer quadratically constrained program. To tackle the nonlinearity in the constraint, we alternatively fix a subset of decision variables and optimize the remaining ones via Mixed Integer Linear Programming (MILP). We solve the MILP problem at each iteration via an off-the-shelf MILP solver. Our experimental results show that our method significantly outperforms existing heuristic methods, employing either independent optimization or joint optimization strategies. We have also verified the generalization ability of our method over grid environments with various sizes and its high robustness to the algorithm hyper-parameters.
△ Less
Submitted 31 January, 2025;
originally announced February 2025.
-
Experimental Evaluation of an SDN Controller for Open Optical-circuit-switched Networks
Authors:
Kazuya Anazawa,
Takeru Inoue,
Toru Mano,
Hiroshi Ou,
Hirotaka Ujikawa,
Dmitrii Briantcev,
Sumaiya Binte Ali,
Devika Dass,
Hideki Nishizawa,
Yoshiaki Sone,
Eoin Kenny,
Marco Ruffini,
Daniel Kilper,
Eiji Oki,
Koichi Takasugi
Abstract:
Open optical networks have been considered to be important for cost-effectively building and operating the networks. Recently, the optical-circuit-switches (OCSes) have attracted industry and academia because of their cost efficiency and higher capacity than traditional electrical packet switches (EPSes) and reconfigurable optical add drop multiplexers (ROADMs). Though the open interfaces and cont…
▽ More
Open optical networks have been considered to be important for cost-effectively building and operating the networks. Recently, the optical-circuit-switches (OCSes) have attracted industry and academia because of their cost efficiency and higher capacity than traditional electrical packet switches (EPSes) and reconfigurable optical add drop multiplexers (ROADMs). Though the open interfaces and control planes for traditional ROADMs and transponders have been defined by several standard-defining organizations (SDOs), those of OCSes have not. Considering that several OCSes have already been installed in production datacenter networks (DCNs) and several OCS products are on the market, bringing the openness and interoperability into the OCS-based networks has become important. Motivated by this fact, this paper investigates a software-defined networking (SDN) controller for open optical-circuit-switched networks. To this end, we identified the use cases of OCSes and derived the controller requirements for supporting them. We then proposed a multi-vendor (MV) OCS controller framework that satisfies the derived requirements; it was designed to quickly and consistently operate fiber paths upon receiving the operation requests. We validated our controller by implementing it and evaluating its performance on actual MV-OCS networks. It satisfied all the requirements, and fiber paths could be configured within 1.0 second by using our controller.
△ Less
Submitted 29 April, 2025; v1 submitted 28 January, 2025;
originally announced January 2025.
-
Routing and Scheduling Optimization for Urban Air Mobility Fleet Management using Quantum Annealing
Authors:
Renichiro Haba,
Takuya Mano,
Ryosuke Ueda,
Genichiro Ebe,
Kohei Takeda,
Masayoshi Terabe,
Masayuki Ohzeki
Abstract:
The growing integration of urban air mobility (UAM) for urban transportation and delivery has accelerated due to increasing traffic congestion and its environmental and economic repercussions. Efficiently managing the anticipated high-density air traffic in cities is critical to ensure safe and effective operations. In this study, we propose a routing and scheduling framework to address the needs…
▽ More
The growing integration of urban air mobility (UAM) for urban transportation and delivery has accelerated due to increasing traffic congestion and its environmental and economic repercussions. Efficiently managing the anticipated high-density air traffic in cities is critical to ensure safe and effective operations. In this study, we propose a routing and scheduling framework to address the needs of a large fleet of UAM vehicles operating in urban areas. Using mathematical optimization techniques, we plan efficient and deconflicted routes for a fleet of vehicles. Formulating route planning as a maximum weighted independent set problem enables us to utilize various algorithms and specialized optimization hardware, such as quantum annealers, which has seen substantial progress in recent years. Our method is validated using a traffic management simulator tailored for the airspace in Singapore. Our approach enhances airspace utilization by distributing traffic throughout a region. This study broadens the potential applications of optimization techniques in UAM traffic management.
△ Less
Submitted 14 October, 2024;
originally announced October 2024.
-
AI Surrogate Model for Distributed Computing Workloads
Authors:
David K. Park,
Yihui Ren,
Ozgur O. Kilic,
Tatiana Korchuganova,
Sairam Sri Vatsavai,
Joseph Boudreau,
Tasnuva Chowdhury,
Shengyu Feng,
Raees Khan,
Jaehyung Kim,
Scott Klasky,
Tadashi Maeno,
Paul Nilsson,
Verena Ingrid Martinez Outschoorn,
Norbert Podhorszki,
Frederic Suter,
Wei Yang,
Yiming Yang,
Shinjae Yoo,
Alexei Klimentov,
Adolfy Hoisie
Abstract:
Large-scale international scientific collaborations, such as ATLAS, Belle II, CMS, and DUNE, generate vast volumes of data. These experiments necessitate substantial computational power for varied tasks, including structured data processing, Monte Carlo simulations, and end-user analysis. Centralized workflow and data management systems are employed to handle these demands, but current decision-ma…
▽ More
Large-scale international scientific collaborations, such as ATLAS, Belle II, CMS, and DUNE, generate vast volumes of data. These experiments necessitate substantial computational power for varied tasks, including structured data processing, Monte Carlo simulations, and end-user analysis. Centralized workflow and data management systems are employed to handle these demands, but current decision-making processes for data placement and payload allocation are often heuristic and disjointed. This optimization challenge potentially could be addressed using contemporary machine learning methods, such as reinforcement learning, which, in turn, require access to extensive data and an interactive environment. Instead, we propose a generative surrogate modeling approach to address the lack of training data and concerns about privacy preservation. We have collected and processed real-world job submission records, totaling more than two million jobs through 150 days, and applied four generative models for tabular data -- TVAE, CTAGGAN+, SMOTE, and TabDDPM -- to these datasets, thoroughly evaluating their performance. Along with measuring the discrepancy among feature-wise distributions separately, we also evaluate pair-wise feature correlations, distance to closest record, and responses to pre-trained models. Our experiments indicate that SMOTE and TabDDPM can generate similar tabular data, almost indistinguishable from the ground truth. Yet, as a non-learning method, SMOTE ranks the lowest in privacy preservation. As a result, we conclude that the probabilistic-diffusion-model-based TabDDPM is the most suitable generative model for managing job record data.
△ Less
Submitted 10 October, 2024;
originally announced October 2024.
-
Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory
Authors:
Edward Karavakis,
Wen Guan,
Zhaoyu Yang,
Tadashi Maeno,
Torre Wenaus,
Jennifer Adelman-McCarthy,
Fernando Barreiro Megino,
Kaushik De,
Richard Dubois,
Michelle Gower,
Tim Jenness,
Alexei Klimentov,
Tatiana Korchuganova,
Mikolaj Kowalik,
Fa-Hui Lin,
Paul Nilsson,
Sergey Padolski,
Wei Yang,
Shuwei Ye
Abstract:
The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment's run. More than 20 terabytes of data must be stored ev…
▽ More
The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment's run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory's needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.
△ Less
Submitted 8 December, 2023;
originally announced December 2023.
-
Dynamic optical path provisioning for alien access links: architecture, demonstration, and challenges
Authors:
Hideki Nishizawa,
Takeo Sasai,
Takeru Inoue,
Kazuya Anazawa,
Toru Mano,
Kei Kitamura,
Yoshiaki Sone,
Tetsuro Inui,
Koichi Takasugi
Abstract:
With the spread of Data Center Interconnect (DCI) and local 5G, there is a growing need for dynamically established connections between customer locations through high-capacity optical links. However, link parameters such as signal power profile and amplifier gains are often unknown and have to be measured by experts, preventing dynamic path provisioning due to the time-consuming manual measuremen…
▽ More
With the spread of Data Center Interconnect (DCI) and local 5G, there is a growing need for dynamically established connections between customer locations through high-capacity optical links. However, link parameters such as signal power profile and amplifier gains are often unknown and have to be measured by experts, preventing dynamic path provisioning due to the time-consuming manual measurements. Although several techniques for estimating the unknown parameters of such alien access links have been proposed, no work has presented architecture and protocol that drive the estimation techniques to establish an optical path between the customer locations. Our study aims to automatically connect customer-owned transceivers via alien access links with optimal quality of transmission (QoT). We first propose an architecture and protocol for cooperative optical path design between a customer and carrier, utilizing a state-of-the-art technique for estimating link parameters. We then implement the proposed protocol in a software-defined network (SDN) controller and white-box transponders using an open API. The experiments demonstrate that the optical path is dynamically established via alien access links in 137 seconds from the transceiver's cold start. Lastly, we discuss the QoT accuracy obtained with this method and the remaining issues.
△ Less
Submitted 12 September, 2022; v1 submitted 6 September, 2022;
originally announced September 2022.
-
An intelligent Data Delivery Service for and beyond the ATLAS experiment
Authors:
Wen Guan,
Tadashi Maeno,
Brian Paul Bockelman,
Torre Wenaus,
Fahui Lin,
Siarhei Padolski,
Rui Zhang,
Aleksandr Alekseev
Abstract:
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. iDDS has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service around a workflow-orien…
▽ More
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. iDDS has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service around a workflow-oriented structure to work with existing and emerging use cases in ATLAS and other experiments. Here we will present the motivation for iDDS, its design schema and architecture, use cases and current status, and plans for the future.
△ Less
Submitted 28 February, 2021;
originally announced March 2021.
-
Towards an Intelligent Data Delivery Service
Authors:
Wen Guan,
Tadashi Maeno,
Gancho Dimitrov,
Brian Paul Bockelman,
Torre Wenaus,
Vakhtang Tsulaia,
Nicolo Magini
Abstract:
The ATLAS Event Streaming Service (ESS) at the LHC is an approach to preprocess and deliver data for Event Service (ES) that has implemented a fine-grained approach for ATLAS event processing. The ESS allows one to asynchronously deliver only the input events required by ES processing, with the aim to decrease data traffic over WAN and improve overall data processing throughput. A prototype of ESS…
▽ More
The ATLAS Event Streaming Service (ESS) at the LHC is an approach to preprocess and deliver data for Event Service (ES) that has implemented a fine-grained approach for ATLAS event processing. The ESS allows one to asynchronously deliver only the input events required by ES processing, with the aim to decrease data traffic over WAN and improve overall data processing throughput. A prototype of ESS was developed to deliver streaming events to fine-grained ES jobs. Based on it, an intelligent Data Delivery Service (iDDS) is under development to decouple the "cold format" and the processing format of the data, which also opens the opportunity to include the production systems of other HEP experiments. Here we will at first present the ESS model view and its motivations for iDDS system. Then we will also present the iDDS schema, architecture and the applications of iDDS.
△ Less
Submitted 3 July, 2020;
originally announced July 2020.
-
Unified System for Processing Real and Simulated Data in the ATLAS Experiment
Authors:
Mikhail Borodin,
Kaushik De,
Jose Garcia Navarro,
Dmitry Golubkov,
Alexei Klimentov,
Tadashi Maeno,
David South,
Alexandre Vaniachine
Abstract:
The physics goals of the next Large Hadron Collider run include high precision tests of the Standard Model and searches for new physics. These goals require detailed comparison of data with computational models simulating the expected data behavior. To highlight the role which modeling and simulation plays in future scientific discovery, we report on use cases and experience with a unified system…
▽ More
The physics goals of the next Large Hadron Collider run include high precision tests of the Standard Model and searches for new physics. These goals require detailed comparison of data with computational models simulating the expected data behavior. To highlight the role which modeling and simulation plays in future scientific discovery, we report on use cases and experience with a unified system built to process both real and simulated data of growing volume and variety.
△ Less
Submitted 28 August, 2015;
originally announced August 2015.
-
Polynomial Expressions of Carries in p-ary Arithmetics
Authors:
Shizuo Kaji,
Toshiaki Maeno,
Koji Nuida,
Yasuhide Numata
Abstract:
It is known that any $n$-variable function on a finite prime field of characteristic $p$ can be expressed as a polynomial over the same field with at most $p^n$ monomials. However, it is not obvious to determine the polynomial for a given concrete function. In this paper, we study the concrete polynomial expressions of the carries in addition and multiplication of $p$-ary integers. For the case of…
▽ More
It is known that any $n$-variable function on a finite prime field of characteristic $p$ can be expressed as a polynomial over the same field with at most $p^n$ monomials. However, it is not obvious to determine the polynomial for a given concrete function. In this paper, we study the concrete polynomial expressions of the carries in addition and multiplication of $p$-ary integers. For the case of addition, our result gives a new family of symmetric polynomials, which generalizes the known result for the binary case $p = 2$ where the carries are given by elementary symmetric polynomials. On the other hand, for the case of multiplication of $n$ single-digit integers, we give a simple formula of the polynomial expression for the carry to the next digit using the Bernoulli numbers, and show that it has only $(n+1)(p-1)/2 + 1$ monomials, which is significantly fewer than the worst-case number $p^n$ of monomials for general functions. We also discuss applications of our results to cryptographic computation on encrypted data.
△ Less
Submitted 18 February, 2016; v1 submitted 8 June, 2015;
originally announced June 2015.
-
MILJS : Brand New JavaScript Libraries for Matrix Calculation and Machine Learning
Authors:
Ken Miura,
Tetsuaki Mano,
Atsushi Kanehira,
Yuichiro Tsuchiya,
Tatsuya Harada
Abstract:
MILJS is a collection of state-of-the-art, platform-independent, scalable, fast JavaScript libraries for matrix calculation and machine learning. Our core library offering a matrix calculation is called Sushi, which exhibits far better performance than any other leading machine learning libraries written in JavaScript. Especially, our matrix multiplication is 177 times faster than the fastest Java…
▽ More
MILJS is a collection of state-of-the-art, platform-independent, scalable, fast JavaScript libraries for matrix calculation and machine learning. Our core library offering a matrix calculation is called Sushi, which exhibits far better performance than any other leading machine learning libraries written in JavaScript. Especially, our matrix multiplication is 177 times faster than the fastest JavaScript benchmark. Based on Sushi, a machine learning library called Tempura is provided, which supports various algorithms widely used in machine learning research. We also provide Soba as a visualization library. The implementations of our libraries are clearly written, properly documented and thus can are easy to get started with, as long as there is a web browser. These libraries are available from http://mil-tokyo.github.io/ under the MIT license.
△ Less
Submitted 20 February, 2015;
originally announced February 2015.
-
A mathematical problem for security analysis of hash functions and pseudorandom generators
Authors:
Koji Nuida,
Takuro Abe,
Shizuo Kaji,
Toshiaki Maeno,
Yasuhide Numata
Abstract:
In this paper, we specify a class of mathematical problems, which we refer to as "Function Density Problems" (FDPs, in short), and point out novel connections of FDPs to the following two cryptographic topics; theoretical security evaluations of keyless hash functions (such as SHA-1), and constructions of provably secure pseudorandom generators (PRGs) with some enhanced security property introduce…
▽ More
In this paper, we specify a class of mathematical problems, which we refer to as "Function Density Problems" (FDPs, in short), and point out novel connections of FDPs to the following two cryptographic topics; theoretical security evaluations of keyless hash functions (such as SHA-1), and constructions of provably secure pseudorandom generators (PRGs) with some enhanced security property introduced by Dubrov and Ishai [STOC 2006]. Our argument aims at proposing new theoretical frameworks for these topics (especially for the former) based on FDPs, rather than providing some concrete and practical results on the topics. We also give some examples of mathematical discussions on FDPs, which would be of independent interest from mathematical viewpoints. Finally, we discuss possible directions of future research on other cryptographic applications of FDPs and on mathematical studies on FDPs themselves.
△ Less
Submitted 28 August, 2014; v1 submitted 31 May, 2012;
originally announced June 2012.
