-
D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Authors:
Rafael Rafailov,
Kyle Hatch,
Anikait Singh,
Laura Smith,
Aviral Kumar,
Ilya Kostrikov,
Philippe Hansen-Estruch,
Victor Kolev,
Philip Ball,
Jiajun Wu,
Chelsea Finn,
Sergey Levine
Abstract:
Offline reinforcement learning algorithms hold the promise of enabling data-driven RL methods that do not require costly or dangerous real-world exploration and benefit from large pre-collected datasets. This in turn can facilitate real-world applications, as well as a more standardized approach to RL research. Furthermore, offline RL methods can provide effective initializations for online finetu…
▽ More
Offline reinforcement learning algorithms hold the promise of enabling data-driven RL methods that do not require costly or dangerous real-world exploration and benefit from large pre-collected datasets. This in turn can facilitate real-world applications, as well as a more standardized approach to RL research. Furthermore, offline RL methods can provide effective initializations for online finetuning to overcome challenges with exploration. However, evaluating progress on offline RL algorithms requires effective and challenging benchmarks that capture properties of real-world tasks, provide a range of task difficulties, and cover a range of challenges both in terms of the parameters of the domain (e.g., length of the horizon, sparsity of rewards) and the parameters of the data (e.g., narrow demonstration data or broad exploratory data). While considerable progress in offline RL in recent years has been enabled by simpler benchmark tasks, the most widely used datasets are increasingly saturating in performance and may fail to reflect properties of realistic tasks. We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments, based on models of real-world robotic systems, and comprising a variety of data sources, including scripted data, play-style data collected by human teleoperators, and other data sources. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation, with some of the tasks specifically designed to require both pre-training and fine-tuning. We hope that our proposed benchmark will facilitate further progress on both offline RL and fine-tuning algorithms. Website with code, examples, tasks, and data is available at \url{https://sites.google.com/view/d5rl/}
△ Less
Submitted 15 August, 2024;
originally announced August 2024.
-
Efficient Imitation Learning with Conservative World Models
Authors:
Victor Kolev,
Rafael Rafailov,
Kyle Hatch,
Jiajun Wu,
Chelsea Finn
Abstract:
We tackle the problem of policy learning from expert demonstrations without a reward function. A central challenge in this space is that these policies fail upon deployment due to issues of distributional shift, environment stochasticity, or compounding errors. Adversarial imitation learning alleviates this issue but requires additional on-policy training samples for stability, which presents a ch…
▽ More
We tackle the problem of policy learning from expert demonstrations without a reward function. A central challenge in this space is that these policies fail upon deployment due to issues of distributional shift, environment stochasticity, or compounding errors. Adversarial imitation learning alleviates this issue but requires additional on-policy training samples for stability, which presents a challenge in realistic domains due to inefficient learning and high sample complexity. One approach to this issue is to learn a world model of the environment, and use synthetic data for policy training. While successful in prior works, we argue that this is sub-optimal due to additional distribution shifts between the learned model and the real environment. Instead, we re-frame imitation learning as a fine-tuning problem, rather than a pure reinforcement learning one. Drawing theoretical connections to offline RL and fine-tuning algorithms, we argue that standard online world model algorithms are not well suited to the imitation learning problem. We derive a principled conservative optimization bound and demonstrate empirically that it leads to improved performance on two very challenging manipulation environments from high-dimensional raw pixel observations. We set a new state-of-the-art performance on the Franka Kitchen environment from images, requiring only 10 demos on no reward labels, as well as solving a complex dexterity manipulation task.
△ Less
Submitted 15 August, 2024; v1 submitted 21 May, 2024;
originally announced May 2024.
-
MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
Authors:
Rafael Rafailov,
Kyle Hatch,
Victor Kolev,
John D. Martin,
Mariano Phielipp,
Chelsea Finn
Abstract:
We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning to either improve the performance of the agent over the data collection policy or adapt to novel tasks. At the same time, model-based RL algorithms have ach…
▽ More
We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning to either improve the performance of the agent over the data collection policy or adapt to novel tasks. At the same time, model-based RL algorithms have achieved significant progress in sample efficiency and the complexity of the tasks they can solve, yet remain under-utilized in the fine-tuning setting. In this work, we argue that existing model-based offline RL methods are not suitable for offline-to-online fine-tuning in high-dimensional domains due to issues with distribution shifts, off-dynamics data, and non-stationary rewards. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization, while preventing model exploitation by controlling epistemic uncertainty. We find that our approach successfully solves tasks from the MetaWorld benchmark, as well as the Franka Kitchen robot manipulation environment completely from images. To the best of our knowledge, MOTO is the first method to solve this environment from pixels.
△ Less
Submitted 6 January, 2024;
originally announced January 2024.
-
Bauer's Spectral Factorization Method for Low Order Multiwavelet Filter Design
Authors:
Vasil Kolev,
Todor Cooklev,
Fritz Keinert
Abstract:
Para-Hermitian polynomial matrices obtained by matrix spectral factorization lead to functions useful in control theory systems, basis functions in numerical methods or multiscaling functions used in signal processing. We introduce a fast algorithm for matrix spectral factorization based on Bauer$'$s method. We convert Bauer$'$ method into a nonlinear matrix equation (NME). The NME is solved by tw…
▽ More
Para-Hermitian polynomial matrices obtained by matrix spectral factorization lead to functions useful in control theory systems, basis functions in numerical methods or multiscaling functions used in signal processing. We introduce a fast algorithm for matrix spectral factorization based on Bauer$'$s method. We convert Bauer$'$ method into a nonlinear matrix equation (NME). The NME is solved by two different numerical algorithms (Fixed Point Iteration and Newton$'$s Method) which produce approximate scalar or matrix factors, as well as a symbolic algorithm which produces exact factors in closed form for some low-order scalar or matrix polynomial matrices, respectively. Convergence rates of the two numerical algorithms are investigated for a number of singular and nonsingular scalar and matrix polynomials taken from different areas. In particular, one of the singular examples leads to new orthogonal multiscaling and multiwavelet filters. Since the NME can also be solved as a Generalized Discrete Time Algebraic Riccati Equation (GDARE), numerical results using built-in routines in Maple 17.0 and 6 Matlab versions are presented.
△ Less
Submitted 8 December, 2023;
originally announced December 2023.
-
Watch Nearby! Privacy Analysis of the People Nearby Service of Telegram
Authors:
Maurantonio Caprolu,
Savio Sciancalepore,
Aleksandar Grigorov,
Velyan Kolev,
Gabriele Oligeri
Abstract:
People Nearby is a service offered by Telegram that allows a user to discover other Telegram users, based only on geographical proximity. Nearby users are reported with a rough estimate of their distance from the position of the reference user, allowing Telegram to claim location privacy In this paper, we systematically analyze the location privacy provided by Telegram to users of the People Nearb…
▽ More
People Nearby is a service offered by Telegram that allows a user to discover other Telegram users, based only on geographical proximity. Nearby users are reported with a rough estimate of their distance from the position of the reference user, allowing Telegram to claim location privacy In this paper, we systematically analyze the location privacy provided by Telegram to users of the People Nearby service. Through an extensive measurement campaign run by spoofing the user's location all over the world, we reverse-engineer the algorithm adopted by People Nearby to compute distances between users. Although the service protects against precise user localization, we demonstrate that location privacy is always lower than the one declared by Telegram of 500 meters. Specifically, we discover that location privacy is a function of the geographical position of the user. Indeed, the radius of the location privacy area (localization error) spans between 400 meters (close to the equator) and 128 meters (close to the poles), with a difference of up to 75% (worst case) compared to what Telegram declares. After our responsible disclosure, Telegram updated the FAQ associated with the service. Finally, we provide some solutions and countermeasures that Telegram can implement to improve location privacy. In general, the reported findings highlight the significant privacy risks associated with using People Nearby service.
△ Less
Submitted 20 October, 2023;
originally announced October 2023.
-
Assessing of Soil Erosion Risk Through Geoinformation Sciences and Remote Sensing -- A Review
Authors:
Lachezar Filchev,
Vasil Kolev
Abstract:
During past decades a marked manifestation of widespread erosion phenomena was studied worldwide. Global conservation community has launched campaigns at local, regional and continental level in developing countries for preservation of soil resources in order not only to stop or mitigate human impact on nature but also to improve life in rural areas introducing new approaches for soil cultivation.…
▽ More
During past decades a marked manifestation of widespread erosion phenomena was studied worldwide. Global conservation community has launched campaigns at local, regional and continental level in developing countries for preservation of soil resources in order not only to stop or mitigate human impact on nature but also to improve life in rural areas introducing new approaches for soil cultivation. After the adoption of Sustainable Development Goals of UNs and launching several world initiatives such as the Land Degradation Neutrality (LDN) the world came to realize the very importance of the soil resources on which the biosphere relies for its existence. The main goal of the chapter is to review different types and structures erosion models as well as their applications. Several methods using spatial analysis capabilities of geographic information systems (GIS) are in operation for soil erosion risk assessment, such as Universal Soil Loss Equation (USLE), Revised Universal Soil Loss Equation (RUSLE) in operation worldwide and in the USA and MESALES model. These and more models are being discussed in the present work alongside more experimental models and methods for assessing soil erosion risk such as Artificial Intelligence (AI), Machine and Deep Learning, etc. At the end of this work, a prospectus for the future development of soil erosion risk assessment is drawn.
△ Less
Submitted 12 October, 2023;
originally announced October 2023.
-
US \& MRI Image Fusion Based on Markerless Skin Registration
Authors:
Martina Paccini,
Giacomo Paschina,
Stefano De Beni,
Andrei Stefanov,
Velizar Kolev,
Giuseppe Patanè
Abstract:
This paper presents an innovative automatic fusion imaging system that combines 3D CT/MR images with real-time ultrasound (US) acquisition. The system eliminates the need for external physical markers and complex training, making image fusion feasible for physicians with different experience levels. The integrated system involves a portable 3D camera for patient-specific surface acquisition, an el…
▽ More
This paper presents an innovative automatic fusion imaging system that combines 3D CT/MR images with real-time ultrasound (US) acquisition. The system eliminates the need for external physical markers and complex training, making image fusion feasible for physicians with different experience levels. The integrated system involves a portable 3D camera for patient-specific surface acquisition, an electromagnetic tracking system, and US components. The fusion algorithm comprises two main parts: skin segmentation and rigid co-registration, both integrated into the US machine. The co-registration software aligns the surface extracted from CT/MR images with patient-specific coordinates, facilitating rapid and effective fusion. Experimental testing in different settings validates the system's accuracy, computational efficiency, noise robustness, and operator independence. The co-registration error remains under the acceptable range of~$1$ cm.
△ Less
Submitted 4 March, 2024; v1 submitted 26 July, 2023;
originally announced July 2023.
-
Wavelet Coherence Of Total Solar Irradiance and Atlantic Climate
Authors:
Vasil Kolev,
Yavor Chapanov
Abstract:
The oscillations of climatic parameters of North Atlantic Ocean play important role in various events in North America and Europe. Several climatic indices are associated with these oscillations. The long term Atlantic temperature anomalies are described by the Atlantic Multidecadal Oscillation (AMO). The Atlantic Multidecadal Oscillation also known as Atlantic Multidecadal Variability (AMV), is t…
▽ More
The oscillations of climatic parameters of North Atlantic Ocean play important role in various events in North America and Europe. Several climatic indices are associated with these oscillations. The long term Atlantic temperature anomalies are described by the Atlantic Multidecadal Oscillation (AMO). The Atlantic Multidecadal Oscillation also known as Atlantic Multidecadal Variability (AMV), is the variability of the sea surface temperature (SST) of the North Atlantic Ocean at the timescale of several decades. The AMO is correlated to air temperatures and rainfall over much of the Northern Hemisphere, in particular in the summer climate in North America and Europe. The long-term variations of surface temperature are driven mainly by the cycles of solar activity, represented by the variations of the Total Solar Irradiance (TSI). The frequency and amplitude dependences between the TSI and AMO are analyzed by wavelet coherence of millennial time series since 800 AD till now. The results of wavelet coherence are compared with the detected common solar and climate cycles in narrow frequency bands by the method of Partial Fourier Approximation. The long-term coherence between TSI and AMO can help to understand better the recent climate change and can improve the long term forecast.
△ Less
Submitted 3 May, 2023;
originally announced May 2023.
-
Challenges and Solutions for Utilizing Earth Observations in the "Big Data" era
Authors:
Lachezar Filchev,
Lyubka Pashova,
Vasil Kolev,
Stuart Frye
Abstract:
The ever-growing need of data preservation and their systematic analysis contributing to sustainable development of the society spurred in the past decade,numerous Big Data projects and initiatives are focusing on the Earth Observation (EO). The number of Big Data EO applications has grown extremely worldwide almost simultaneously with other scientific and technological areas of the human knowledg…
▽ More
The ever-growing need of data preservation and their systematic analysis contributing to sustainable development of the society spurred in the past decade,numerous Big Data projects and initiatives are focusing on the Earth Observation (EO). The number of Big Data EO applications has grown extremely worldwide almost simultaneously with other scientific and technological areas of the human knowledge due to the revolutionary technological progress in the space and information technology sciences. The substantial contribution to this development are the space programs of the renowned space agencies, such as NASA, ESA,Roskosmos, JAXA, DLR, INPE, ISRO, CNES etc. A snap-shot of the current Big Data sets from available satellite missions covering the Bulgarian territory is also presented. This short overview of the geoscience Big Data collection with a focus on EO will emphasize to the multiple Vs of EO in order to provide a snapshot on the current state-of-the-art in EO data preservation and manipulation. Main modern approaches for compressing, clustering and modelling EO in the geoinformation science for Big Data analysis, interpretation and visualization for a variety of applications are outlined. Special attention is paid to the contemporary EO data modelling and visualization systems.
△ Less
Submitted 19 August, 2021;
originally announced August 2021.
-
Neural Abstract Reasoner
Authors:
Victor Kolev,
Bogdan Georgiev,
Svetlin Penkov
Abstract:
Abstract reasoning and logic inference are difficult problems for neural networks, yet essential to their applicability in highly structured domains. In this work we demonstrate that a well known technique such as spectral regularization can significantly boost the capabilities of a neural learner. We introduce the Neural Abstract Reasoner (NAR), a memory augmented architecture capable of learning…
▽ More
Abstract reasoning and logic inference are difficult problems for neural networks, yet essential to their applicability in highly structured domains. In this work we demonstrate that a well known technique such as spectral regularization can significantly boost the capabilities of a neural learner. We introduce the Neural Abstract Reasoner (NAR), a memory augmented architecture capable of learning and using abstract rules. We show that, when trained with spectral regularization, NAR achieves $78.8\%$ accuracy on the Abstraction and Reasoning Corpus, improving performance 4 times over the best known human hand-crafted symbolic solvers. We provide some intuition for the effects of spectral regularization in the domain of abstract reasoning based on theoretical generalization bounds and Solomonoff's theory of inductive inference.
△ Less
Submitted 12 November, 2020;
originally announced November 2020.
-
Design of a Simple Orthogonal Multiwavelet Filter by Matrix Spectral Factorization
Authors:
Vasil Kolev,
Todor Cooklev,
Fritz Keinert
Abstract:
We consider the design of an orthogonal symmetric/antisymmetric multiwavelet from its matrix product filter by matrix spectral factorization (MSF). As a test problem, we construct a simple matrix product filter with desirable properties, and factor it using Bauer's method, which in this case can be done in closed form. The corresponding orthogonal multiwavelet function is derived using algebraic t…
▽ More
We consider the design of an orthogonal symmetric/antisymmetric multiwavelet from its matrix product filter by matrix spectral factorization (MSF). As a test problem, we construct a simple matrix product filter with desirable properties, and factor it using Bauer's method, which in this case can be done in closed form. The corresponding orthogonal multiwavelet function is derived using algebraic techniques which allow symmetry to be considered. This leads to the known orthogonal multiwavelet SA1, which can also be derived directly. We also give a lifting scheme for SA1, investigate the influence of the number of significant digits in the calculations, and show some numerical experiments.
△ Less
Submitted 18 August, 2021; v1 submitted 15 October, 2019;
originally announced October 2019.
-
Matrix Spectral Factorization for SA4 Multiwavelet
Authors:
Vasil Kolev,
Todor Cooklev,
Fritz Keinert
Abstract:
In this paper, we investigate Bauer's method for the matrix spectral factorization of an r-channel matrix product filter which is a halfband autocorrelation matrix. We regularize the resulting matrix spectral factors by an averaging approach and by multiplication by a unitary matrix. This leads to the approximate and exact orthogonal SA4 multiscaling functions. We also find the corresponding ortho…
▽ More
In this paper, we investigate Bauer's method for the matrix spectral factorization of an r-channel matrix product filter which is a halfband autocorrelation matrix. We regularize the resulting matrix spectral factors by an averaging approach and by multiplication by a unitary matrix. This leads to the approximate and exact orthogonal SA4 multiscaling functions. We also find the corresponding orthogonal multiwavelet functions, based on the QR decomposition.
△ Less
Submitted 18 August, 2021; v1 submitted 14 October, 2019;
originally announced October 2019.
-
Singular Value Decomposition of Images from Scanned Photographic Plates
Authors:
Vasil Kolev,
Katya Tsvetkova,
Milcho Tsvetkov
Abstract:
We want to approximate the mxn image A from scanned astronomical photographic plates (from the Sofia Sky Archive Data Center) by using far fewer entries than in the original matrix. By using rank of a matrix, k we remove the redundant information or noise and use as Wiener filter, when rank k<m or k<n. With this approximation more than 98% compression ration of image of astronomical plate without…
▽ More
We want to approximate the mxn image A from scanned astronomical photographic plates (from the Sofia Sky Archive Data Center) by using far fewer entries than in the original matrix. By using rank of a matrix, k we remove the redundant information or noise and use as Wiener filter, when rank k<m or k<n. With this approximation more than 98% compression ration of image of astronomical plate without that image details, is obtained. The SVD of images from scanned photographic plates (SPP) is considered and its possible image compression.
△ Less
Submitted 18 August, 2021; v1 submitted 7 October, 2013;
originally announced October 2013.
-
Compressed sensing of astronomical images:orthogonal wavelets domains
Authors:
Vasil Kolev
Abstract:
A simple approach for orthogonal wavelets in compressed sensing (CS) applications is presented. We compare efficient algorithm for different orthogonal wavelet measurement matrices in CS for image processing from scanned photographic plates (SPP). Some important characteristics were obtained for astronomical image processing of SPP. The best orthogonal wavelet choice for measurement matrix constru…
▽ More
A simple approach for orthogonal wavelets in compressed sensing (CS) applications is presented. We compare efficient algorithm for different orthogonal wavelet measurement matrices in CS for image processing from scanned photographic plates (SPP). Some important characteristics were obtained for astronomical image processing of SPP. The best orthogonal wavelet choice for measurement matrix construction in CS for image compression of images of SPP is given. The image quality measure for linear and nonlinear image compression method is defined.
△ Less
Submitted 18 August, 2021; v1 submitted 27 November, 2011;
originally announced November 2011.
-
Multiplierless Modules for Forward and Backward Integer Wavelet Transform
Authors:
Vasil Kolev
Abstract:
This article is about the architecture of a lossless wavelet filter bank with reprogrammable logic. It is based on second generation of wavelets with a reduced of number of operations. A new basic structure for parallel architecture and modules to forward and backward integer discrete wavelet transform is proposed.
This article is about the architecture of a lossless wavelet filter bank with reprogrammable logic. It is based on second generation of wavelets with a reduced of number of operations. A new basic structure for parallel architecture and modules to forward and backward integer discrete wavelet transform is proposed.
△ Less
Submitted 18 August, 2021; v1 submitted 19 October, 2010;
originally announced October 2010.
-
Orthogonal multifilters image processing of astronomical images from scanned photographic plates
Authors:
Vasil Kolev
Abstract:
In this paper orthogonal multifilters for astronomical image processing are presented. We obtained new orthogonal multifilters based on the orthogonal wavelet of Haar and Daubechies. Recently, multiwavelets have been introduced as a more powerful multiscale analysis tool. It adds several degrees of freedom in multifilter design and makes it possible to have several useful properties such as symmet…
▽ More
In this paper orthogonal multifilters for astronomical image processing are presented. We obtained new orthogonal multifilters based on the orthogonal wavelet of Haar and Daubechies. Recently, multiwavelets have been introduced as a more powerful multiscale analysis tool. It adds several degrees of freedom in multifilter design and makes it possible to have several useful properties such as symmetry, orthogonality, short support, and a higher number of vanishing moments simultaneously. Multifilter decomposition of scanned photographic plates with astronomical images is made.
△ Less
Submitted 18 August, 2021; v1 submitted 22 July, 2010;
originally announced July 2010.
