-
A MARL-based Approach for Easing MAS Organization Engineering
Authors:
Julien Soulé,
Jean-Paul Jamont,
Michel Occello,
Louis-Marie Traonouez,
Paul Théron
Abstract:
Multi-Agent Systems (MAS) have been successfully applied in industry for their ability to address complex, distributed problems, especially in IoT-based systems. Their efficiency in achieving given objectives and meeting design requirements is strongly dependent on the MAS organization during the engineering process of an application-specific MAS. To design a MAS that can achieve given goals, avai…
▽ More
Multi-Agent Systems (MAS) have been successfully applied in industry for their ability to address complex, distributed problems, especially in IoT-based systems. Their efficiency in achieving given objectives and meeting design requirements is strongly dependent on the MAS organization during the engineering process of an application-specific MAS. To design a MAS that can achieve given goals, available methods rely on the designer's knowledge of the deployment environment. However, high complexity and low readability in some deployment environments make the application of these methods to be costly or raise safety concerns. In order to ease the MAS organization design regarding those concerns, we introduce an original Assisted MAS Organization Engineering Approach (AOMEA). AOMEA relies on combining a Multi-Agent Reinforcement Learning (MARL) process with an organizational model to suggest relevant organizational specifications to help in MAS engineering.
△ Less
Submitted 5 June, 2025;
originally announced June 2025.
-
Towards a Multi-Agent Simulation of Cyber-attackers and Cyber-defenders Battles
Authors:
Julien Soulé,
Jean-Paul Jamont,
Michel Occello,
Paul Théron,
Louis-Marie Traonouez
Abstract:
As cyber-attacks show to be more and more complex and coordinated, cyber-defenders strategy through multi-agent approaches could be key to tackle against cyber-attacks as close as entry points in a networked system. This paper presents a Markovian modeling and implementation through a simulator of fighting cyber-attacker agents and cyber-defender agents deployed on host network nodes. It aims to p…
▽ More
As cyber-attacks show to be more and more complex and coordinated, cyber-defenders strategy through multi-agent approaches could be key to tackle against cyber-attacks as close as entry points in a networked system. This paper presents a Markovian modeling and implementation through a simulator of fighting cyber-attacker agents and cyber-defender agents deployed on host network nodes. It aims to provide an experimental framework to implement realistically based coordinated cyber-attack scenarios while assessing cyber-defenders dynamic organizations. We abstracted network nodes by sets of properties including agents' ones. Actions applied by agents model how the network reacts depending in a given state and what properties are to change. Collective choice of the actions brings the whole environment closer or farther from respective cyber-attackers and cyber-defenders goals. Using the simulator, we implemented a realistically inspired scenario with several behavior implementation approaches for cyber-defenders and cyber-attackers.
△ Less
Submitted 5 June, 2025;
originally announced June 2025.
-
Streamlining Resilient Kubernetes Autoscaling with Multi-Agent Systems via an Automated Online Design Framework
Authors:
Julien Soulé,
Jean-Paul Jamont,
Michel Occello,
Louis-Marie Traonouez,
Paul Théron
Abstract:
In cloud-native systems, Kubernetes clusters with interdependent services often face challenges to their operational resilience due to poor workload management issues such as resource blocking, bottlenecks, or continuous pod crashes. These vulnerabilities are further amplified in adversarial scenarios, such as Distributed Denial-of-Service attacks (DDoS). Conventional Horizontal Pod Autoscaling (H…
▽ More
In cloud-native systems, Kubernetes clusters with interdependent services often face challenges to their operational resilience due to poor workload management issues such as resource blocking, bottlenecks, or continuous pod crashes. These vulnerabilities are further amplified in adversarial scenarios, such as Distributed Denial-of-Service attacks (DDoS). Conventional Horizontal Pod Autoscaling (HPA) approaches struggle to address such dynamic conditions, while reinforcement learning-based methods, though more adaptable, typically optimize single goals like latency or resource usage, neglecting broader failure scenarios. We propose decomposing the overarching goal of maintaining operational resilience into failure-specific sub-goals delegated to collaborative agents, collectively forming an HPA Multi-Agent System (MAS). We introduce an automated, four-phase online framework for HPA MAS design: 1) modeling a digital twin built from cluster traces; 2) training agents in simulation using roles and missions tailored to failure contexts; 3) analyzing agent behaviors for explainability; and 4) transferring learned policies to the real cluster. Experimental results demonstrate that the generated HPA MASs outperform three state-of-the-art HPA systems in sustaining operational resilience under various adversarial conditions in a proposed complex cluster.
△ Less
Submitted 26 May, 2025;
originally announced May 2025.
-
An Organizationally-Oriented Approach to Enhancing Explainability and Control in Multi-Agent Reinforcement Learning
Authors:
Julien Soulé,
Jean-Paul Jamont,
Michel Occello,
Louis-Marie Traonouez,
Paul Théron
Abstract:
Multi-Agent Reinforcement Learning can lead to the development of collaborative agent behaviors that show similarities with organizational concepts. Pushing forward this perspective, we introduce a novel framework that explicitly incorporates organizational roles and goals from the $\mathcal{M}OISE^+$ model into the MARL process, guiding agents to satisfy corresponding organizational constraints.…
▽ More
Multi-Agent Reinforcement Learning can lead to the development of collaborative agent behaviors that show similarities with organizational concepts. Pushing forward this perspective, we introduce a novel framework that explicitly incorporates organizational roles and goals from the $\mathcal{M}OISE^+$ model into the MARL process, guiding agents to satisfy corresponding organizational constraints. By structuring training with roles and goals, we aim to enhance both the explainability and control of agent behaviors at the organizational level, whereas much of the literature primarily focuses on individual agents. Additionally, our framework includes a post-training analysis method to infer implicit roles and goals, offering insights into emergent agent behaviors. This framework has been applied across various MARL environments and algorithms, demonstrating coherence between predefined organizational specifications and those inferred from trained agents.
△ Less
Submitted 30 March, 2025;
originally announced March 2025.
-
Doers, not Watchers: Intelligent Autonomous Agents are a Path to Cyber Resilience
Authors:
Alexander Kott,
Paul Theron
Abstract:
Today's cyber defense tools are mostly watchers. They are not active doers. To be sure, watching too is a demanding affair. These tools monitor the traffic and events; they detect malicious signatures, patterns and anomalies; they might classify and characterize what they observe; they issue alerts, and they might even learn while doing all this. But they don't act. They do little to plan and exec…
▽ More
Today's cyber defense tools are mostly watchers. They are not active doers. To be sure, watching too is a demanding affair. These tools monitor the traffic and events; they detect malicious signatures, patterns and anomalies; they might classify and characterize what they observe; they issue alerts, and they might even learn while doing all this. But they don't act. They do little to plan and execute responses to attacks, and they don't plan and execute recovery activities. Response and recovery - core elements of cyber resilience are left to the human cyber analysts, incident responders and system administrators. We believe things should change. Cyber defense tools should not be merely watchers. They need to become doers - active fighters in maintaining a system's resilience against cyber threats. This means that their capabilities should include a significant degree of autonomy and intelligence for the purposes of rapid response to a compromise - either incipient or already successful - and rapid recovery that aids the resilience of the overall system. Often, the response and recovery efforts need to be undertaken in absence of any human involvement, and with an intelligent consideration of risks and ramifications of such efforts. Recently an international team published a report that proposes a vision of an autonomous intelligent cyber defense agent (AICA) and offers a high-level reference architecture of such an agent. In this paper we explore this vision.
△ Less
Submitted 26 January, 2022;
originally announced January 2022.
-
When Autonomous Intelligent Goodware will Fight Autonomous Intelligent Malware: A Possible Future of Cyber Defense
Authors:
Paul Théron,
Alexander Kott
Abstract:
In the coming years, the future of military combat will include, on one hand, artificial intelligence-optimized complex command, control, communications, computers, intelligence, surveillance and reconnaissance (C4ISR) and networks and, on the other hand, autonomous intelligent Things fighting autonomous intelligent Things at a fast pace. Under this perspective, enemy forces will seek to disable o…
▽ More
In the coming years, the future of military combat will include, on one hand, artificial intelligence-optimized complex command, control, communications, computers, intelligence, surveillance and reconnaissance (C4ISR) and networks and, on the other hand, autonomous intelligent Things fighting autonomous intelligent Things at a fast pace. Under this perspective, enemy forces will seek to disable or disturb our autonomous Things and our complex infrastructures and systems. Autonomy, scale and complexity in our defense systems will trigger new cyber-attack strategies, and autonomous intelligent malware (AIM) will be part of the picture. Should these cyber-attacks succeed while human operators remain unaware or unable to react fast enough due to the speed, scale or complexity of the mission, systems or attacks, missions would fail, our networks and C4ISR would be heavily disrupted, and command and control would be disabled. New cyber-defense doctrines and technologies are therefore required. Autonomous cyber defense (ACyD) is a new field of research and technology driven by the defense sector in anticipation of such threats to future military infrastructures, systems and operations. It will be implemented via swarms of autonomous intelligent cyber-defense agents (AICAs) that will fight AIM within our networks and systems. This paper presents this cyber-defense technology of the future, the current state of the art in this field and its main challenges. First, we review the rationale of the ACyD concept and its associated AICA technology. Then, we present the current research results from NATO's IST-152 Research Task Group on the AICA Reference Architecture. We then develop the 12 main technological challenges that must be resolved in the coming years, besides ethical and political issues.
△ Less
Submitted 25 November, 2019;
originally announced December 2019.
-
Towards an Active, Autonomous and Intelligent Cyber Defense of Military Systems: the NATO AICA Reference Architecture
Authors:
Paul Theron,
Alexander Kott,
Martin Drašar,
Krzysztof Rzadca,
Benoît LeBlanc,
Mauno Pihelgas,
Luigi Mancini,
Agostino Panico
Abstract:
Within the future Global Information Grid, complex massively interconnected systems, isolated defense vehicles, sensors and effectors, and infrastructures and systems demanding extremely low failure rates, to which human security operators cannot have an easy access and cannot deliver fast enough reactions to cyber-attacks, need an active, autonomous and intelligent cyber defense. Multi Agent Syst…
▽ More
Within the future Global Information Grid, complex massively interconnected systems, isolated defense vehicles, sensors and effectors, and infrastructures and systems demanding extremely low failure rates, to which human security operators cannot have an easy access and cannot deliver fast enough reactions to cyber-attacks, need an active, autonomous and intelligent cyber defense. Multi Agent Systems for Cyber Defense may provide an answer to this requirement. This paper presents the concept and architecture of an Autonomous Intelligent Cyber defense Agent (AICA). First, we describe the rationale of the AICA concept. Secondly, we explain the methodology and purpose that drive the definition of the AICA Reference Architecture (AICARA) by NATO's IST-152 Research and Technology Group. Thirdly, we review some of the main features and challenges of Multi Autonomous Intelligent Cyber defense Agent (MAICA). Fourthly, we depict the initially assumed AICA Reference Architecture. Then we present one of our preliminary research issues, assumptions and ideas. Finally, we present the future lines of research that will help develop and test the AICA / MAICA concept.
△ Less
Submitted 7 June, 2018;
originally announced June 2018.
-
Toward Intelligent Autonomous Agents for Cyber Defense: Report of the 2017 Workshop by the North Atlantic Treaty Organization (NATO) Research Group IST-152-RTG
Authors:
Alexander Kott,
Ryan Thomas,
Martin Drašar,
Markus Kont,
Alex Poylisher,
Benjamin Blakely,
Paul Theron,
Nathaniel Evans,
Nandi Leslie,
Rajdeep Singh,
Maria Rigaki,
S Jay Yang,
Benoit LeBlanc,
Paul Losiewicz,
Sylvain Hourlier,
Misty Blowers,
Hugh Harney,
Gregory Wehner,
Alessandro Guarino,
Jana Komárková,
James Rowell
Abstract:
This report summarizes the discussions and findings of the Workshop on Intelligent Autonomous Agents for Cyber Defence and Resilience organized by the NATO research group IST-152-RTG. The workshop was held in Prague, Czech Republic, on 18-20 October 2017. There is a growing recognition that future cyber defense should involve extensive use of partially autonomous agents that actively patrol the fr…
▽ More
This report summarizes the discussions and findings of the Workshop on Intelligent Autonomous Agents for Cyber Defence and Resilience organized by the NATO research group IST-152-RTG. The workshop was held in Prague, Czech Republic, on 18-20 October 2017. There is a growing recognition that future cyber defense should involve extensive use of partially autonomous agents that actively patrol the friendly network, and detect and react to hostile activities rapidly (far faster than human reaction time), before the hostile malware is able to inflict major damage, evade friendly agents, or destroy friendly agents. This requires cyber-defense agents with a significant degree of intelligence, autonomy, self-learning, and adaptability. The report focuses on the following questions: In what computing and tactical environments would such an agent operate? What data would be available for the agent to observe or ingest? What actions would the agent be able to take? How would such an agent plan a complex course of actions? Would the agent learn from its experiences, and how? How would the agent collaborate with humans? How can we ensure that the agent will not take undesirable destructive actions? Is it possible to help envision such an agent with a simple example?
△ Less
Submitted 20 April, 2018;
originally announced April 2018.
-
Autonomous Intelligent Cyber-defense Agent (AICA) Reference Architecture. Release 2.0
Authors:
Alexander Kott,
Paul Théron,
Martin Drašar,
Edlira Dushku,
Benoît LeBlanc,
Paul Losiewicz,
Alessandro Guarino,
Luigi Mancini,
Agostino Panico,
Mauno Pihelgas,
Krzysztof Rzadca,
Fabio De Gaspari
Abstract:
This report - a major revision of its previous release - describes a reference architecture for intelligent software agents performing active, largely autonomous cyber-defense actions on military networks of computing and communicating devices. The report is produced by the North Atlantic Treaty Organization (NATO) Research Task Group (RTG) IST-152 "Intelligent Autonomous Agents for Cyber Defense…
▽ More
This report - a major revision of its previous release - describes a reference architecture for intelligent software agents performing active, largely autonomous cyber-defense actions on military networks of computing and communicating devices. The report is produced by the North Atlantic Treaty Organization (NATO) Research Task Group (RTG) IST-152 "Intelligent Autonomous Agents for Cyber Defense and Resilience". In a conflict with a technically sophisticated adversary, NATO military tactical networks will operate in a heavily contested battlefield. Enemy software cyber agents - malware - will infiltrate friendly networks and attack friendly command, control, communications, computers, intelligence, surveillance, and reconnaissance and computerized weapon systems. To fight them, NATO needs artificial cyber hunters - intelligent, autonomous, mobile agents specialized in active cyber defense. With this in mind, in 2016, NATO initiated RTG IST-152. Its objective has been to help accelerate the development and transition to practice of such software agents by producing a reference architecture and technical roadmap. This report presents the concept and architecture of an Autonomous Intelligent Cyber-defense Agent (AICA). We describe the rationale of the AICA concept, explain the methodology and purpose that drive the definition of the AICA Reference Architecture, and review some of the main features and challenges of AICAs.
△ Less
Submitted 22 March, 2023; v1 submitted 28 March, 2018;
originally announced March 2018.
