-
Coordination of Electrical and Heating Resources by Self-Interested Agents
Authors:
Rico Schrage,
Jari Radler,
Astrid Nieße
Abstract:
With the rise of distributed energy resources and sector coupling, distributed optimization can be a sensible approach to coordinate decentralized energy resources. Further, district heating, heat pumps, cogeneration, and sharing concepts like local energy communities introduce the potential to optimize heating and electricity output simultaneously. To solve this issue, we tackle the distributed m…
▽ More
With the rise of distributed energy resources and sector coupling, distributed optimization can be a sensible approach to coordinate decentralized energy resources. Further, district heating, heat pumps, cogeneration, and sharing concepts like local energy communities introduce the potential to optimize heating and electricity output simultaneously. To solve this issue, we tackle the distributed multi-energy scheduling optimization problem, which describes the optimization of distributed energy generators over multiple time steps to reach a specific target schedule. This work describes a novel distributed hybrid algorithm as a solution approach. This approach is based on the heuristics of gossiping and local search and can simultaneously optimize the private objective of the participants and the collective objective, considering multiple energy sectors. We show that the algorithm finds globally near-optimal solutions while protecting the stakeholders' economic goals and the plants' technical properties. Two test cases representing pure electrical and gas-based technologies are evaluated.
△ Less
Submitted 19 June, 2025;
originally announced June 2025.
-
A General Approach of Automated Environment Design for Learning the Optimal Power Flow
Authors:
Thomas Wolgast,
Astrid Nieße
Abstract:
Reinforcement learning (RL) algorithms are increasingly used to solve the optimal power flow (OPF) problem. Yet, the question of how to design RL environments to maximize training performance remains unanswered, both for the OPF and the general case. We propose a general approach for automated RL environment design by utilizing multi-objective optimization. For that, we use the hyperparameter opti…
▽ More
Reinforcement learning (RL) algorithms are increasingly used to solve the optimal power flow (OPF) problem. Yet, the question of how to design RL environments to maximize training performance remains unanswered, both for the OPF and the general case. We propose a general approach for automated RL environment design by utilizing multi-objective optimization. For that, we use the hyperparameter optimization (HPO) framework, which allows the reuse of existing HPO algorithms and methods. On five OPF benchmark problems, we demonstrate that our automated design approach consistently outperforms a manually created baseline environment design. Further, we use statistical analyses to determine which environment design decisions are especially important for performance, resulting in multiple novel insights on how RL-OPF environments should be designed. Finally, we discuss the risk of overfitting the environment to the utilized RL algorithm. To the best of our knowledge, this is the first general approach for automated RL environment design.
△ Less
Submitted 1 May, 2025;
originally announced May 2025.
-
Ten Recommendations for Engineering Research Software in Energy Research
Authors:
Stephan Ferenz,
Emilie Frost,
Rico Schrage,
Thomas Wolgast,
Inga Beyers,
Oliver Karras,
Oliver Werth,
Astrid Nieße
Abstract:
Energy research software (ERS) is a central cornerstone to facilitate energy research. However, ERS is developed by researchers who, in many cases, lack formal training in software engineering. This reduces the quality of ERS, leading to limited reproducibility and reusability. To address these issues, we developed ten central recommendations for the development of ERS, covering areas such as conc…
▽ More
Energy research software (ERS) is a central cornerstone to facilitate energy research. However, ERS is developed by researchers who, in many cases, lack formal training in software engineering. This reduces the quality of ERS, leading to limited reproducibility and reusability. To address these issues, we developed ten central recommendations for the development of ERS, covering areas such as conceptualization, development, testing, and publication of ERS. The recommendations are based on the outcomes of two workshops with a diverse group of energy researchers and aim to improve the awareness of research software engineering in the energy domain. The recommendations should enhance the quality of ERS and, therefore, the reproducibility of energy research.
△ Less
Submitted 19 February, 2025;
originally announced February 2025.
-
Learning the Optimal Power Flow: Environment Design Matters
Authors:
Thomas Wolgast,
Astrid Nieße
Abstract:
To solve the optimal power flow (OPF) problem, reinforcement learning (RL) emerges as a promising new approach. However, the RL-OPF literature is strongly divided regarding the exact formulation of the OPF problem as an RL environment. In this work, we collect and implement diverse environment design decisions from the literature regarding training data, observation space, episode definition, and…
▽ More
To solve the optimal power flow (OPF) problem, reinforcement learning (RL) emerges as a promising new approach. However, the RL-OPF literature is strongly divided regarding the exact formulation of the OPF problem as an RL environment. In this work, we collect and implement diverse environment design decisions from the literature regarding training data, observation space, episode definition, and reward function choice. In an experimental analysis, we show the significant impact of these environment design options on RL-OPF training performance. Further, we derive some first recommendations regarding the choice of these design decisions. The created environment framework is fully open-source and can serve as a benchmark for future research in the RL-OPF field.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
A Metadata-Based Ecosystem to Improve the FAIRness of Research Software
Authors:
Patrick Kuckertz,
Jan Göpfert,
Oliver Karras,
David Neuroth,
Julian Schönau,
Rodrigo Pueblas,
Stephan Ferenz,
Felix Engel,
Noah Pflugradt,
Jann M. Weinand,
Astrid Nieße,
Sören Auer,
Detlef Stolten
Abstract:
The reuse of research software is central to research efficiency and academic exchange. The application of software enables researchers with varied backgrounds to reproduce, validate, and expand upon study findings. Furthermore, the analysis of open source code aids in the comprehension, comparison, and integration of approaches. Often, however, no further use occurs because relevant software cann…
▽ More
The reuse of research software is central to research efficiency and academic exchange. The application of software enables researchers with varied backgrounds to reproduce, validate, and expand upon study findings. Furthermore, the analysis of open source code aids in the comprehension, comparison, and integration of approaches. Often, however, no further use occurs because relevant software cannot be found or is incompatible with existing research processes. This results in repetitive software development, which impedes the advancement of individual researchers and entire research communities. In this article, the DataDesc ecosystem is presented, an approach to describing data models of software interfaces with detailed and machine-actionable metadata. In addition to a specialized metadata schema, an exchange format and support tools for easy collection and the automated publishing of software documentation are introduced. This approach practically increases the FAIRness, i.e., findability, accessibility, interoperability, and so the reusability of research software, as well as effectively promotes its impact on research.
△ Less
Submitted 18 June, 2023;
originally announced June 2023.
-
ANALYSE -- Learning to Attack Cyber-Physical Energy Systems With Intelligent Agents
Authors:
Thomas Wolgast,
Nils Wenninghoff,
Stephan Balduin,
Eric Veith,
Bastian Fraune,
Torben Woltjen,
Astrid Nieße
Abstract:
The ongoing penetration of energy systems with information and communications technology (ICT) and the introduction of new markets increase the potential for malicious or profit-driven attacks that endanger system stability. To ensure security-of-supply, it is necessary to analyze such attacks and their underlying vulnerabilities, to develop countermeasures and improve system design. We propose AN…
▽ More
The ongoing penetration of energy systems with information and communications technology (ICT) and the introduction of new markets increase the potential for malicious or profit-driven attacks that endanger system stability. To ensure security-of-supply, it is necessary to analyze such attacks and their underlying vulnerabilities, to develop countermeasures and improve system design. We propose ANALYSE, a machine-learning-based software suite to let learning agents autonomously find attacks in cyber-physical energy systems, consisting of the power system, ICT, and energy markets. ANALYSE is a modular, configurable, and self-documenting framework designed to find yet unknown attack types and to reproduce many known attack strategies in cyber-physical energy systems from the scientific literature.
△ Less
Submitted 21 April, 2023;
originally announced May 2023.
-
Approximating Energy Market Clearing and Bidding With Model-Based Reinforcement Learning
Authors:
Thomas Wolgast,
Astrid Nieße
Abstract:
Energy market rules should incentivize market participants to behave in a market and grid conform way. However, they can also provide incentives for undesired and unexpected strategies if the market design is flawed. Multi-agent Reinforcement learning (MARL) is a promising new approach to predicting the expected profit-maximizing behavior of energy market participants in simulation. However, reinf…
▽ More
Energy market rules should incentivize market participants to behave in a market and grid conform way. However, they can also provide incentives for undesired and unexpected strategies if the market design is flawed. Multi-agent Reinforcement learning (MARL) is a promising new approach to predicting the expected profit-maximizing behavior of energy market participants in simulation. However, reinforcement learning requires many interactions with the system to converge, and the power system environment often consists of extensive computations, e.g., optimal power flow (OPF) calculation for market clearing. To tackle this complexity, we provide a model of the energy market to a basic MARL algorithm in the form of a learned OPF approximation and explicit market rules. The learned OPF surrogate model makes an explicit solving of the OPF completely unnecessary. Our experiments demonstrate that the model additionally reduces training time by about one order of magnitude but at the cost of a slightly worse performance. Potential applications of our method are market design, more realistic modeling of market participants, and analysis of manipulative behavior.
△ Less
Submitted 1 November, 2023; v1 submitted 3 March, 2023;
originally announced March 2023.
-
A Multi-Criteria Metaheuristic Algorithm for Distributed Optimization of Electric Energy Storage
Authors:
Rico Schrage,
Paul Hendrik Tiemann,
Astrid Nieße
Abstract:
The distributed schedule optimization of energy storage constitutes a challenge. Such algorithms often expect an input set containing all feasible schedules or respectively require to efficiently search the schedule space. It is hardly possible to accomplish this with energy storage due to its high flexibility. In this paper, the problem is introduced in detail and addressed by a metaheuristic alg…
▽ More
The distributed schedule optimization of energy storage constitutes a challenge. Such algorithms often expect an input set containing all feasible schedules or respectively require to efficiently search the schedule space. It is hardly possible to accomplish this with energy storage due to its high flexibility. In this paper, the problem is introduced in detail and addressed by a metaheuristic algorithm, which generates a preselection of schedules. Three contributions are presented to achieve this goal: First, an extension for a distributed schedule optimization allowing a simultaneous optimization is developed. Second, an evolutionary algorithm is designed to generate optimized schedules. Third, the algorithm is extended to include an arbitrary local criterion. It is shown that the presented approach is suitable to schedule electric energy storage in real households and industries with different generator and storage types.
△ Less
Submitted 4 November, 2022; v1 submitted 15 August, 2022;
originally announced August 2022.
-
Dynamic communication topologies for distributed heuristics in energy system optimization algorithms
Authors:
Stefanie Holly,
Astrid Nieße
Abstract:
The communication topology is an essential aspect in designing distributed optimization heuristics. It can influence the exploration and exploitation of the search space and thus the optimization performance in terms of solution quality, convergence speed and collaboration costs, all relevant aspects for applications operating critical infrastructure in energy systems. In this work, we present an…
▽ More
The communication topology is an essential aspect in designing distributed optimization heuristics. It can influence the exploration and exploitation of the search space and thus the optimization performance in terms of solution quality, convergence speed and collaboration costs, all relevant aspects for applications operating critical infrastructure in energy systems. In this work, we present an approach for adapting the communication topology during runtime, based on the principles of simulated annealing. We compare the approach to common static topologies regarding the performance of an exemplary distributed optimization heuristic. Finally, we investigate the correlations between fitness landscape properties and defined performance metrics.
△ Less
Submitted 3 August, 2021;
originally announced August 2021.
-
Analyzing Power Grid, ICT, and Market Without Domain Knowledge Using Distributed Artificial Intelligence
Authors:
Eric MSP Veith,
Stephan Balduin,
Nils Wenninghoff,
Martin Tröschel,
Lars Fischer,
Astrid Nieße,
Thomas Wolgast,
Richard Sethmann,
Bastian Fraune,
Torben Woltjen
Abstract:
Modern cyber-physical systems (CPS), such as our energy infrastructure, are becoming increasingly complex: An ever-higher share of Artificial Intelligence (AI)-based technologies use the Information and Communication Technology (ICT) facet of energy systems for operation optimization, cost efficiency, and to reach CO2 goals worldwide. At the same time, markets with increased flexibility and ever s…
▽ More
Modern cyber-physical systems (CPS), such as our energy infrastructure, are becoming increasingly complex: An ever-higher share of Artificial Intelligence (AI)-based technologies use the Information and Communication Technology (ICT) facet of energy systems for operation optimization, cost efficiency, and to reach CO2 goals worldwide. At the same time, markets with increased flexibility and ever shorter trade horizons enable the multi-stakeholder situation that is emerging in this setting. These systems still form critical infrastructures that need to perform with highest reliability. However, today's CPS are becoming too complex to be analyzed in the traditional monolithic approach, where each domain, e.g., power grid and ICT as well as the energy market, are considered as separate entities while ignoring dependencies and side-effects. To achieve an overall analysis, we introduce the concept for an application of distributed artificial intelligence as a self-adaptive analysis tool that is able to analyze the dependencies between domains in CPS by attacking them. It eschews pre-configured domain knowledge, instead exploring the CPS domains for emergent risk situations and exploitable loopholes in codices, with a focus on rational market actors that exploit the system while still following the market rules.
△ Less
Submitted 10 June, 2020;
originally announced June 2020.
-
Analyzing Cyber-Physical Systems from the Perspective of Artificial Intelligence
Authors:
Eric M. S. P. Veith,
Lars Fischer,
Martin Tröschel,
Astrid Nieße
Abstract:
Principles of modern cyber-physical system (CPS) analysis are based on analytical methods that depend on whether safety or liveness requirements are considered. Complexity is abstracted through different techniques, ranging from stochastic modelling to contracts. However, both distributed heuristics and Artificial Intelligence (AI)-based approaches as well as the user perspective or unpredictable…
▽ More
Principles of modern cyber-physical system (CPS) analysis are based on analytical methods that depend on whether safety or liveness requirements are considered. Complexity is abstracted through different techniques, ranging from stochastic modelling to contracts. However, both distributed heuristics and Artificial Intelligence (AI)-based approaches as well as the user perspective or unpredictable effects, such as accidents or the weather, introduce enough uncertainty to warrant reinforcement-learning-based approaches. This paper compares traditional approaches in the domain of CPS modelling and analysis with the AI researcher perspective to exploring unknown complex systems.
△ Less
Submitted 21 August, 2019;
originally announced August 2019.
