-
Toward Explainable Offline RL: Analyzing Representations in Intrinsically Motivated Decision Transformers
Authors:
Leonardo Guiducci,
Antonio Rizzo,
Giovanna Maria Dimitri
Abstract:
Elastic Decision Transformers (EDTs) have proved to be particularly successful in offline reinforcement learning, offering a flexible framework that unifies sequence modeling with decision-making under uncertainty. Recent research has shown that incorporating intrinsic motivation mechanisms into EDTs improves performance across exploration tasks, yet the representational mechanisms underlying thes…
▽ More
Elastic Decision Transformers (EDTs) have proved to be particularly successful in offline reinforcement learning, offering a flexible framework that unifies sequence modeling with decision-making under uncertainty. Recent research has shown that incorporating intrinsic motivation mechanisms into EDTs improves performance across exploration tasks, yet the representational mechanisms underlying these improvements remain unexplored. In this paper, we introduce a systematic post-hoc explainability framework to analyze how intrinsic motivation shapes learned embeddings in EDTs. Through statistical analysis of embedding properties (including covariance structure, vector magnitudes, and orthogonality), we reveal that different intrinsic motivation variants create fundamentally different representational structures. Our analysis demonstrates environment-specific correlation patterns between embedding metrics and performance that explain why intrinsic motivation improves policy learning. These findings show that intrinsic motivation operates beyond simple exploration bonuses, acting as a representational prior that shapes embedding geometry in biologically plausible ways, creating environment-specific organizational structures that facilitate better decision-making.
△ Less
Submitted 16 June, 2025;
originally announced June 2025.
-
Reinforcement Learning Based Prediction of PID Controller Gains for Quadrotor UAVs
Authors:
Serhat Sönmez,
Luca Montecchio,
Simone Martini,
Matthew J. Rutherford,
Alessandro Rizzo,
Margareta Stefanovic,
Kimon P. Valavanis
Abstract:
A reinforcement learning (RL) based methodology is proposed and implemented for online fine-tuning of PID controller gains, thus, improving quadrotor effective and accurate trajectory tracking. The RL agent is first trained offline on a quadrotor PID attitude controller and then validated through simulations and experimental flights. RL exploits a Deep Deterministic Policy Gradient (DDPG) algorith…
▽ More
A reinforcement learning (RL) based methodology is proposed and implemented for online fine-tuning of PID controller gains, thus, improving quadrotor effective and accurate trajectory tracking. The RL agent is first trained offline on a quadrotor PID attitude controller and then validated through simulations and experimental flights. RL exploits a Deep Deterministic Policy Gradient (DDPG) algorithm, which is an off-policy actor-critic method. Training and simulation studies are performed using Matlab/Simulink and the UAV Toolbox Support Package for PX4 Autopilots. Performance evaluation and comparison studies are performed between the hand-tuned and RL-based tuned approaches. The results show that the controller parameters based on RL are adjusted during flights, achieving the smallest attitude errors, thus significantly improving attitude tracking performance compared to the hand-tuned approach.
△ Less
Submitted 6 February, 2025;
originally announced February 2025.
-
Benchmarking Quantum Convolutional Neural Networks for Signal Classification in Simulated Gamma-Ray Burst Detection
Authors:
Farida Farsian,
Nicolò Parmiggiani,
Alessandro Rizzo,
Gabriele Panebianco,
Andrea Bulgarelli,
Francesco Schillirò,
Carlo Burigana,
Vincenzo Cardone,
Luca Cappelli,
Massimo Meneghetti,
Giuseppe Murante,
Giuseppe Sarracino,
Roberto Scaramella,
Vincenzo Testa,
Tiziana Trombetti
Abstract:
This study evaluates the use of Quantum Convolutional Neural Networks (QCNNs) for identifying signals resembling Gamma-Ray Bursts (GRBs) within simulated astrophysical datasets in the form of light curves. The task addressed here focuses on distinguishing GRB-like signals from background noise in simulated Cherenkov Telescope Array Observatory (CTAO) data, the next-generation astrophysical observa…
▽ More
This study evaluates the use of Quantum Convolutional Neural Networks (QCNNs) for identifying signals resembling Gamma-Ray Bursts (GRBs) within simulated astrophysical datasets in the form of light curves. The task addressed here focuses on distinguishing GRB-like signals from background noise in simulated Cherenkov Telescope Array Observatory (CTAO) data, the next-generation astrophysical observatory for very high-energy gamma-ray science. QCNNs, a quantum counterpart of classical Convolutional Neural Networks (CNNs), leverage quantum principles to process and analyze high-dimensional data efficiently. We implemented a hybrid quantum-classical machine learning technique using the Qiskit framework, with the QCNNs trained on a quantum simulator. Several QCNN architectures were tested, employing different encoding methods such as Data Reuploading and Amplitude encoding. Key findings include that QCNNs achieved accuracy comparable to classical CNNs, often surpassing 90\%, while using fewer parameters, potentially leading to more efficient models in terms of computational resources. A benchmark study further examined how hyperparameters like the number of qubits and encoding methods affected performance, with more qubits and advanced encoding methods generally enhancing accuracy but increasing complexity. QCNNs showed robust performance on time-series datasets, successfully detecting GRB signals with high precision. The research is a pioneering effort in applying QCNNs to astrophysics, offering insights into their potential and limitations. This work sets the stage for future investigations to fully realize the advantages of QCNNs in astrophysical data analysis.
△ Less
Submitted 28 January, 2025;
originally announced January 2025.
-
Quantum Convolutional Neural Networks for the detection of Gamma-Ray Bursts in the AGILE space mission data
Authors:
A. Rizzo,
N. Parmiggiani,
A. Bulgarelli,
A. Macaluso,
V. Fioretti,
L. Castaldini,
A. Di Piano,
G. Panebianco,
C. Pittori,
M. Tavani,
C. Sartori,
C. Burigana,
V. Cardone,
F. Farsian,
M. Meneghetti,
G. Murante,
R. Scaramella,
F. Schillirò,
V. Testa,
T. Trombetti
Abstract:
Quantum computing represents a cutting-edge frontier in artificial intelligence. It makes use of hybrid quantum-classical computation which tries to leverage quantum mechanic principles that allow us to use a different approach to deep learning classification problems. The work presented here falls within the context of the AGILE space mission, launched in 2007 by the Italian Space Agency. We impl…
▽ More
Quantum computing represents a cutting-edge frontier in artificial intelligence. It makes use of hybrid quantum-classical computation which tries to leverage quantum mechanic principles that allow us to use a different approach to deep learning classification problems. The work presented here falls within the context of the AGILE space mission, launched in 2007 by the Italian Space Agency. We implement different Quantum Convolutional Neural Networks (QCNN) that analyze data acquired by the instruments onboard AGILE to detect Gamma-Ray Bursts from sky maps or light curves. We use several frameworks such as TensorFlow-Quantum, Qiskit and PennyLane to simulate a quantum computer. We achieved an accuracy of 95.1% on sky maps with QCNNs, while the classical counterpart achieved 98.8% on the same data, using however hundreds of thousands more parameters.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
Massively Scalable Wavelength Diverse Integrated Photonic Linear Neuron
Authors:
Matthew van Niekerk,
Anthony Rizzo,
Hector Rubio Rivera,
Gerald Leake,
Daniel Coleman,
Christopher Tison,
Michael Fanto,
Keren Bergman,
Stefan Preble
Abstract:
As computing resource demands continue to escalate in the face of big data, cloud-connectivity and the internet of things, it has become imperative to develop new low-power, scalable architectures. Neuromorphic photonics, or photonic neural networks, have become a feasible solution for the physical implementation of efficient algorithms directly on-chip. This application is primarily due to the li…
▽ More
As computing resource demands continue to escalate in the face of big data, cloud-connectivity and the internet of things, it has become imperative to develop new low-power, scalable architectures. Neuromorphic photonics, or photonic neural networks, have become a feasible solution for the physical implementation of efficient algorithms directly on-chip. This application is primarily due to the linear nature of light and the scalability of silicon photonics, specifically leveraging the wide-scale complementary metal-oxide-semiconductor (CMOS) manufacturing infrastructure used to fabricate microelectronics chips. Current neuromorphic photonic implementations stem from two paradigms: wavelength coherent and incoherent. Here, we introduce a novel architecture that supports coherent and incoherent operation to increase the capability and capacity of photonic neural networks with a dramatic reduction in footprint compared to previous demonstrations. As a proof-of-principle, we experimentally demonstrate simple addition and subtraction operations on a foundry-fabricated silicon photonic chip. Additionally, we experimentally validate an on-chip network to predict the logical 2-bit gates AND, OR, and XOR to accuracies of $96.8\%, 99\%,$ and $98.5\%$, respectively. This architecture is compatible with highly wavelength parallel sources, enabling massively scalable photonic neural networks.
△ Less
Submitted 25 August, 2022; v1 submitted 11 May, 2022;
originally announced May 2022.
-
Game-theoretical trajectory planning enhances social acceptability for humans
Authors:
Giada Galati,
Stefano Primatesta,
Sergio Grammatico,
Simone Macrì,
Alessandro Rizzo
Abstract:
Since humans and robots are increasingly sharing portions of their operational spaces, experimental evidence is needed to ascertain the safety and social acceptability of robots in human-populated environments. Although several studies have aimed at devising strategies for robot trajectory planning to perform \emph{safe} motion in populated environments, a few efforts have \emph{measured} to what…
▽ More
Since humans and robots are increasingly sharing portions of their operational spaces, experimental evidence is needed to ascertain the safety and social acceptability of robots in human-populated environments. Although several studies have aimed at devising strategies for robot trajectory planning to perform \emph{safe} motion in populated environments, a few efforts have \emph{measured} to what extent a robot trajectory is \emph{accepted} by humans. Here, we present a navigation system for autonomous robotics that ensures safety and social acceptability of robotic trajectories. We overcome the typical reactive nature of state-of-the-art trajectory planners by leveraging non-cooperative game theory to design a planner that encapsulates human-like features of preservation of a vital space, recognition of groups, sequential and strategized decision making, and smooth obstacle avoidance. Social acceptability is measured through a variation of the Turing test administered in the form of a survey questionnaire to a pool of 691 participants. Comparison terms for our tests are a state-of-the-art navigation algorithm (Enhanced Vector Field Histogram, VFH) and purely human trajectories. While all participants easily recognized the non-human nature of VFH-generated trajectories, the distinction between game-theoretical trajectories and human ones were hardly revealed. These results mark a strong milestone toward the full integration of robots in social environments.
△ Less
Submitted 29 March, 2022;
originally announced March 2022.
-
Reachability analysis in stochastic directed graphs by reinforcement learning
Authors:
Corrado Possieri,
Mattia Frasca,
Alessandro Rizzo
Abstract:
We characterize the reachability probabilities in stochastic directed graphs by means of reinforcement learning methods. In particular, we show that the dynamics of the transition probabilities in a stochastic digraph can be modeled via a difference inclusion, which, in turn, can be interpreted as a Markov decision process. Using the latter framework, we offer a methodology to design reward functi…
▽ More
We characterize the reachability probabilities in stochastic directed graphs by means of reinforcement learning methods. In particular, we show that the dynamics of the transition probabilities in a stochastic digraph can be modeled via a difference inclusion, which, in turn, can be interpreted as a Markov decision process. Using the latter framework, we offer a methodology to design reward functions to provide upper and lower bounds on the reachability probabilities of a set of nodes for stochastic digraphs. The effectiveness of the proposed technique is demonstrated by application to the diffusion of epidemic diseases over time-varying contact networks generated by the proximity patterns of mobile agents.
△ Less
Submitted 25 February, 2022;
originally announced February 2022.
-
Game-theoretic modeling of collective decision-making during epidemics
Authors:
Mengbin Ye,
Lorenzo Zino,
Alessandro Rizzo,
Ming Cao
Abstract:
The spreading dynamics of an epidemic and the collective behavioral pattern of the population over which it spreads are deeply intertwined and the latter can critically shape the outcome of the former. Motivated by this, we design a parsimonious game-theoretic behavioral--epidemic model, in which an interplay of realistic factors shapes the co-evolution of individual decision-making and epidemics…
▽ More
The spreading dynamics of an epidemic and the collective behavioral pattern of the population over which it spreads are deeply intertwined and the latter can critically shape the outcome of the former. Motivated by this, we design a parsimonious game-theoretic behavioral--epidemic model, in which an interplay of realistic factors shapes the co-evolution of individual decision-making and epidemics on a network. Although such a co-evolution is deeply intertwined in the real-world, existing models schematize population behavior as instantaneously reactive, thus being unable to capture human behavior in the long term. Our model offers a unified framework to model and predict complex emergent phenomena, including successful collective responses, periodic oscillations, and resurgent epidemic outbreaks. The framework also allows to assess the effectiveness of different policy interventions on ensuring a collective response that successfully eradicates the outbreak. Two case studies, inspired by real-world diseases, are presented to illustrate the potentialities of the proposed model.
△ Less
Submitted 19 July, 2021; v1 submitted 5 August, 2020;
originally announced August 2020.
-
End-to-End Models for the Analysis of System 1 and System 2 Interactions based on Eye-Tracking Data
Authors:
Alessandro Rossi,
Sara Ermini,
Dario Bernabini,
Dario Zanca,
Marino Todisco,
Alessandro Genovese,
Antonio Rizzo
Abstract:
While theories postulating a dual cognitive system take hold, quantitative confirmations are still needed to understand and identify interactions between the two systems or conflict events. Eye movements are among the most direct markers of the individual attentive load and may serve as an important proxy of information. In this work we propose a computational method, within a modified visual vers…
▽ More
While theories postulating a dual cognitive system take hold, quantitative confirmations are still needed to understand and identify interactions between the two systems or conflict events. Eye movements are among the most direct markers of the individual attentive load and may serve as an important proxy of information. In this work we propose a computational method, within a modified visual version of the well-known Stroop test, for the identification of different tasks and potential conflicts events between the two systems through the collection and processing of data related to eye movements. A statistical analysis shows that the selected variables can characterize the variation of attentive load within different scenarios. Moreover, we show that Machine Learning techniques allow to distinguish between different tasks with a good classification accuracy and to investigate more in depth the gaze dynamics.
△ Less
Submitted 3 February, 2020;
originally announced February 2020.
-
Distributed Estimation of State and Parameters in Multi-Agent Cooperative Load Manipulation
Authors:
Antonio Franchi,
Antonio Petitti,
Alessandro Rizzo
Abstract:
We present two distributed methods for the estimation of the kinematic parameters, the dynamic parameters, and the kinematic state of an unknown planar body manipulated by a decentralized multi-agent system. The proposed approaches rely on the rigid body kinematics and dynamics, on nonlinear observation theory, and on consensus algorithms. The only three requirements are that each agent can exert…
▽ More
We present two distributed methods for the estimation of the kinematic parameters, the dynamic parameters, and the kinematic state of an unknown planar body manipulated by a decentralized multi-agent system. The proposed approaches rely on the rigid body kinematics and dynamics, on nonlinear observation theory, and on consensus algorithms. The only three requirements are that each agent can exert a 2D wrench on the load, it can measure the velocity of its contact point, and that the communication graph is connected. Both theoretical nonlinear observability analysis and convergence proofs are provided. The first method assumes constant parameters while the second one can deal with time-varying parameters and can be applied in parallel to any task-oriented control law. For the cases in which a control law is not provided, we propose a distributed and safe control strategy satisfying the observability condition. The effectiveness and robustness of the estimation strategy is showcased by means of realistic MonteCarlo simulations.
△ Less
Submitted 23 September, 2018; v1 submitted 4 February, 2016;
originally announced February 2016.
