-
Carbon-Aware Temporal Data Transfer Scheduling Across Cloud Datacenters
Authors:
Elvis Rodrigues,
Jacob Goldverg,
Tevfik Kosar
Abstract:
Inter-datacenter communication is a significant part of cloud operations and produces a substantial amount of carbon emissions for cloud data centers, where the environmental impact has already been a pressing issue. In this paper, we present a novel carbon-aware temporal data transfer scheduling framework, called LinTS, which promises to significantly reduce the carbon emission of data transfers…
▽ More
Inter-datacenter communication is a significant part of cloud operations and produces a substantial amount of carbon emissions for cloud data centers, where the environmental impact has already been a pressing issue. In this paper, we present a novel carbon-aware temporal data transfer scheduling framework, called LinTS, which promises to significantly reduce the carbon emission of data transfers between cloud data centers. LinTS produces a competitive transfer schedule and makes scaling decisions, outperforming common heuristic algorithms. LinTS can lower carbon emissions during inter-datacenter transfers by up to 66% compared to the worst case and up to 15% compared to other solutions while preserving all deadline constraints.
△ Less
Submitted 4 June, 2025;
originally announced June 2025.
-
Optimizing Data Transfer Performance and Energy Efficiency with Deep Reinforcement Learning
Authors:
Hasubil Jamil,
Jacob Goldverg,
Elvis Rodrigues,
MD S Q Zulkar Nine,
Tevfik Kosar
Abstract:
The rapid growth of data across fields of science and industry has increased the need to improve the performance of end-to-end data transfers while using the resources more efficiently. In this paper, we present a dynamic, multiparameter reinforcement learning (RL) framework that adjusts application-layer transfer settings during data transfers on shared networks. Our method strikes a balance betw…
▽ More
The rapid growth of data across fields of science and industry has increased the need to improve the performance of end-to-end data transfers while using the resources more efficiently. In this paper, we present a dynamic, multiparameter reinforcement learning (RL) framework that adjusts application-layer transfer settings during data transfers on shared networks. Our method strikes a balance between high throughput and low energy utilization by employing reward signals that focus on both energy efficiency and fairness. The RL agents can pause and resume transfer threads as needed, pausing during heavy network use and resuming when resources are available, to prevent overload and save energy. We evaluate several RL techniques and compare our solution with state-of-the-art methods by measuring computational overhead, adaptability, throughput, and energy consumption. Our experiments show up to 25% increase in throughput and up to 40% reduction in energy usage at the end systems compared to baseline methods, highlighting a fair and energy-efficient way to optimize data transfers in shared network environments.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Carbon-Aware End-to-End Data Movement
Authors:
Jacob Goldverg,
Hasibul Jamil,
Elvis Rodriguez,
Tevfik Kosar
Abstract:
The latest trends in the adoption of cloud, edge, and distributed computing, as well as a rise in applying AI/ML workloads, have created a need to measure, monitor, and reduce the carbon emissions of these compute-intensive workloads and the associated communication costs. The data movement over networks has considerable carbon emission that has been neglected due to the difficulty in measuring th…
▽ More
The latest trends in the adoption of cloud, edge, and distributed computing, as well as a rise in applying AI/ML workloads, have created a need to measure, monitor, and reduce the carbon emissions of these compute-intensive workloads and the associated communication costs. The data movement over networks has considerable carbon emission that has been neglected due to the difficulty in measuring the carbon footprint of a given end-to-end network path. We present a novel network carbon footprint measuring mechanism and propose three ways in which users can optimize scheduling network-intensive tasks to enable carbon savings through shifting tasks in time, space, and overlay networks based on the geographic carbon intensity.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
A Reinforcement Learning Approach to Optimize Available Network Bandwidth Utilization
Authors:
Hasibul Jamil,
Elvis Rodrigues,
Jacob Goldverg,
Tevfik Kosar
Abstract:
Efficient data transfers over high-speed, long-distance shared networks require proper utilization of available network bandwidth. Using parallel TCP streams enables an application to utilize network parallelism and can improve transfer throughput; however, finding the optimum number of parallel TCP streams is challenging due to nondeterministic background traffic sharing the same network. Additio…
▽ More
Efficient data transfers over high-speed, long-distance shared networks require proper utilization of available network bandwidth. Using parallel TCP streams enables an application to utilize network parallelism and can improve transfer throughput; however, finding the optimum number of parallel TCP streams is challenging due to nondeterministic background traffic sharing the same network. Additionally, the non-stationary, multi-objectiveness, and partially-observable nature of network signals in the host systems add extra complexity in finding the current network condition. In this work, we present a novel approach to finding the optimum number of parallel TCP streams using deep reinforcement learning (RL). We devise a learning-based algorithm capable of generalizing different network conditions and utilizing the available network bandwidth intelligently. Contrary to rule-based heuristics that do not generalize well in unknown network scenarios, our RL-based solution can dynamically discover and adapt the parallel TCP stream numbers to maximize the network bandwidth utilization without congesting the network and ensure fairness among contending transfers. We extensively evaluated our RL-based algorithm's performance, comparing it with several state-of-the-art online optimization algorithms. The results show that our RL-based algorithm can find near-optimal solutions 40% faster while achieving up to 15% higher throughput. We also show that, unlike a greedy algorithm, our devised RL-based algorithm can avoid network congestion and fairly share the available network resources among contending transfers.
△ Less
Submitted 30 November, 2022; v1 submitted 21 November, 2022;
originally announced November 2022.
-
Energy-Efficient Data Transfer Optimization via Decision-Tree Based Uncertainty Reduction
Authors:
Hasibul Jamil,
Lavone Rodolph,
Jacob Goldverg,
Tevfik Kosar
Abstract:
The increase and rapid growth of data produced by scientific instruments, the Internet of Things (IoT), and social media is causing data transfer performance and resource consumption to garner much attention in the research community. The network infrastructure and end systems that enable this extensive data movement use a substantial amount of electricity, measured in terawatt-hours per year. Man…
▽ More
The increase and rapid growth of data produced by scientific instruments, the Internet of Things (IoT), and social media is causing data transfer performance and resource consumption to garner much attention in the research community. The network infrastructure and end systems that enable this extensive data movement use a substantial amount of electricity, measured in terawatt-hours per year. Managing energy consumption within the core networking infrastructure is an active research area, but there is a limited amount of work on reducing power consumption at the end systems during active data transfers. This paper presents a novel two-phase dynamic throughput and energy optimization model that utilizes an offline decision-search-tree based clustering technique to encapsulate and categorize historical data transfer log information and an online search optimization algorithm to find the best application and kernel layer parameter combination to maximize the achieved data transfer throughput while minimizing the energy consumption. Our model also incorporates an ensemble method to reduce aleatoric uncertainty in finding optimal application and kernel layer parameters during the offline analysis phase. The experimental evaluation results show that our decision-tree based model outperforms the state-of-the-art solutions in this area by achieving 117% higher throughput on average and also consuming 19% less energy at the end systems during active data transfers.
△ Less
Submitted 24 April, 2022; v1 submitted 15 April, 2022;
originally announced April 2022.
