-
Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles
Authors:
Ruoqi Zhang,
Ziwei Luo,
Jens Sjölund,
Per Mattsson,
Linus Gisslén,
Alessandro Sestini
Abstract:
Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning.…
▽ More
Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning. In this paper, we present CPQE (Consistency Policy with Q-Ensembles), which combines consistency models with Q-ensembles to address these challenges.CPQE leverages uncertainty estimation through Q-ensembles to provide more reliable value function approximations, resulting in better training stability and improved performance compared to classic double Q-network methods. Our extensive experiments across multiple game scenarios demonstrate that CPQE achieves inference speeds of up to 60 Hz -- a significant improvement over state-of-the-art diffusion policies that operate at only 20 Hz -- while maintaining comparable performance to multi-step diffusion approaches. CPQE consistently outperforms state-of-the-art consistency model approaches, showing both higher rewards and enhanced training stability throughout the learning process. These results indicate that CPQE offers a practical solution for deploying diffusion-based policies in games and other real-time applications where both multi-modal behavior modeling and rapid inference are critical requirements.
△ Less
Submitted 21 March, 2025;
originally announced March 2025.
-
Probabilistic Bubble Roadmap
Authors:
Bernhard Wullt,
Mikael Norrlöf,
Per Mattsson,
Thomas B. Schön
Abstract:
Finding a collision-free path is a fundamental problem in robotics, where the sampling based planners have a long line of success. However, this approach is computationally expensive, due to the frequent use of collision-detection. Furthermore, the produced paths are usually jagged and require further post-processing before they can be tracked. Due to their high computational cost, these planners…
▽ More
Finding a collision-free path is a fundamental problem in robotics, where the sampling based planners have a long line of success. However, this approach is computationally expensive, due to the frequent use of collision-detection. Furthermore, the produced paths are usually jagged and require further post-processing before they can be tracked. Due to their high computational cost, these planners are usually restricted to static settings, since they are not able to cope with rapid changes in the environment. In our work, we remove this restriction by introducing a learned signed distance function expressed in the configuration space of the robot. The signed distance allows us to form collision-free spherical regions in the configuration space, which we use to suggest a new multi-query path planner that also works in dynamic settings. We propose the probabilistic bubble roadmap planner, which enhances the probabilistic roadmap planner (PRM) by using spheres as vertices and compute the edges by checking for neighboring spheres which intersect. We benchmark our approach in a static setting where we show that we can produce paths that are shorter than the paths produced by the PRM, while having a smaller sized roadmap and finding the paths faster. Finally, we show that we can rapidly rewire the graph in the case of new obstacles introduced at run time and therefore produce paths in the case of moving obstacles.
△ Less
Submitted 22 February, 2025;
originally announced February 2025.
-
EDHOC is a New Security Handshake Standard: An Overview of Security Analysis
Authors:
Elsa López Pérez,
Inria Göran Selander,
John Preuß Mattsson,
Thomas Watteyne,
Mališa Vučinić
Abstract:
The paper wraps up the call for formal analysis of the new security handshake protocol EDHOC by providing an overview of the protocol as it was standardized, a summary of the formal security analyses conducted by the community, and a discussion on open venues for future work.
The paper wraps up the call for formal analysis of the new security handshake protocol EDHOC by providing an overview of the protocol as it was standardized, a summary of the formal security analyses conducted by the community, and a discussion on open venues for future work.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
Authors:
Ruoqi Zhang,
Ziwei Luo,
Jens Sjölund,
Thomas B. Schön,
Per Mattsson
Abstract:
This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that…
▽ More
This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble.
△ Less
Submitted 8 January, 2025; v1 submitted 6 February, 2024;
originally announced February 2024.
-
Structured state-space models are deep Wiener models
Authors:
Fabio Bonassi,
Carl Andersson,
Per Mattsson,
Thomas B. Schön
Abstract:
The goal of this paper is to provide a system identification-friendly introduction to the Structured State-space Models (SSMs). These models have become recently popular in the machine learning community since, owing to their parallelizability, they can be efficiently and scalably trained to tackle extremely-long sequence classification and regression problems. Interestingly, SSMs appear as an eff…
▽ More
The goal of this paper is to provide a system identification-friendly introduction to the Structured State-space Models (SSMs). These models have become recently popular in the machine learning community since, owing to their parallelizability, they can be efficiently and scalably trained to tackle extremely-long sequence classification and regression problems. Interestingly, SSMs appear as an effective way to learn deep Wiener models, which allows to reframe SSMs as an extension of a model class commonly used in system identification. In order to stimulate a fruitful exchange of ideas between the machine learning and system identification communities, we deem it useful to summarize the recent contributions on the topic in a structured and accessible form. At last, we highlight future research directions for which this community could provide impactful contributions.
△ Less
Submitted 20 May, 2024; v1 submitted 11 December, 2023;
originally announced December 2023.
-
Aiding reinforcement learning for set point control
Authors:
Ruoqi Zhang,
Per Mattsson,
Torbjörn Wigren
Abstract:
While reinforcement learning has made great improvements, state-of-the-art algorithms can still struggle with seemingly simple set-point feedback control problems. One reason for this is that the learned controller may not be able to excite the system dynamics well enough initially, and therefore it can take a long time to get data that is informative enough to learn for good control. The paper co…
▽ More
While reinforcement learning has made great improvements, state-of-the-art algorithms can still struggle with seemingly simple set-point feedback control problems. One reason for this is that the learned controller may not be able to excite the system dynamics well enough initially, and therefore it can take a long time to get data that is informative enough to learn for good control. The paper contributes by augmentation of reinforcement learning with a simple guiding feedback controller, for example, a proportional controller. The key advantage in set point control is a much improved excitation that improves the convergence properties of the reinforcement learning controller significantly. This can be very important in real-world control where quick and accurate convergence is needed. The proposed method is evaluated with simulation and on a real-world double tank process with promising results.
△ Less
Submitted 20 April, 2023;
originally announced April 2023.
-
Robust nonlinear set-point control with reinforcement learning
Authors:
Ruoqi Zhang,
Per Mattsson,
Torbjörn Wigren
Abstract:
There has recently been an increased interest in reinforcement learning for nonlinear control problems. However standard reinforcement learning algorithms can often struggle even on seemingly simple set-point control problems. This paper argues that three ideas can improve reinforcement learning methods even for highly nonlinear set-point control problems: 1) Make use of a prior feedback controlle…
▽ More
There has recently been an increased interest in reinforcement learning for nonlinear control problems. However standard reinforcement learning algorithms can often struggle even on seemingly simple set-point control problems. This paper argues that three ideas can improve reinforcement learning methods even for highly nonlinear set-point control problems: 1) Make use of a prior feedback controller to aid amplitude exploration. 2) Use integrated errors. 3) Train on model ensembles. Together these ideas lead to more efficient training, and a trained set-point controller that is more robust to modelling errors and thus can be directly deployed to real-world nonlinear systems. The claim is supported by experiments with a real-world nonlinear cascaded tank process and a simulated strongly nonlinear pH-control system.
△ Less
Submitted 20 April, 2023;
originally announced April 2023.
-
Quantum-Resistant Cryptography
Authors:
John Preuß Mattsson,
Ben Smeets,
Erik Thormarker
Abstract:
Quantum-resistant cryptography is cryptography that aims to deliver cryptographic functions and protocols that remain secure even if large-scale fault-tolerant quantum computers are built. NIST will soon announce the first selected public-key cryptography algorithms in its Post-Quantum Cryptography (PQC) standardization which is the most important current effort in the field of quantum-resistant c…
▽ More
Quantum-resistant cryptography is cryptography that aims to deliver cryptographic functions and protocols that remain secure even if large-scale fault-tolerant quantum computers are built. NIST will soon announce the first selected public-key cryptography algorithms in its Post-Quantum Cryptography (PQC) standardization which is the most important current effort in the field of quantum-resistant cryptography. This report provides an overview to security experts who do not yet have a deep understanding of quantum-resistant cryptography. It surveys the computational model of quantum computers; the quantum algorithms that affect cryptography the most; the risk of Cryptographically Relevant Quantum Computers (CRQCs) being built; the security of symmetric and public-key cryptography in the presence of CRQCs; the NIST PQC standardization effort; the migration to quantum-resistant public-key cryptography; the relevance of Quantum Key Distribution as a complement to conventional cryptography; and the relevance of Quantum Random Number Generators as a complement to current hardware Random Number Generators.
△ Less
Submitted 1 December, 2021;
originally announced December 2021.
-
Nori: Concealing the Concealed Identifier in 5G
Authors:
John Preuß Mattsson,
Prajwol Kumar Nakarmi
Abstract:
IMSI catchers have been a long standing and serious privacy problem in pre-5G mobile networks. To tackle this, 3GPP introduced the Subscription Concealed Identifier (SUCI) and other countermeasures in 5G. In this paper, we analyze the new SUCI mechanism and discover that it provides very poor anonymity when used with the variable length Network Specific Identifiers (NSI), which are part of the 5G…
▽ More
IMSI catchers have been a long standing and serious privacy problem in pre-5G mobile networks. To tackle this, 3GPP introduced the Subscription Concealed Identifier (SUCI) and other countermeasures in 5G. In this paper, we analyze the new SUCI mechanism and discover that it provides very poor anonymity when used with the variable length Network Specific Identifiers (NSI), which are part of the 5G standard. When applied to real-world name length data, we see that SUCI only provides 1-anonymity, meaning that individual subscribers can easily be identified and tracked. We strongly recommend 3GPP and GSMA to standardize and recommend the use of a padding mechanism for SUCI before variable length identifiers get more commonly used. We further show that the padding schemes, commonly used for network traffic, are not optimal for padding of identifiers based on real names. We propose a new improved padding scheme that achieves much less message expansion for a given $k$-anonymity.
△ Less
Submitted 14 June, 2021; v1 submitted 21 May, 2021;
originally announced May 2021.
-
Automatic Observer Script for StarCraft: Brood War Bot Games (technical report)
Authors:
Björn Persson Mattsson,
Tomáš Vajda,
Michal Čertický
Abstract:
This short report describes an automated BWAPI-based script developed for live streams of a StarCraft Brood War bot tournament, SSCAIT. The script controls the in-game camera in order to follow the relevant events and improve the viewer experience. We enumerate its novel features and provide a few implementation notes.
This short report describes an automated BWAPI-based script developed for live streams of a StarCraft Brood War bot tournament, SSCAIT. The script controls the in-game camera in order to follow the relevant events and improve the viewer experience. We enumerate its novel features and provide a few implementation notes.
△ Less
Submitted 1 May, 2015;
originally announced May 2015.
