-
RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
Authors:
Pranav Atreya,
Karl Pertsch,
Tony Lee,
Moo Jin Kim,
Arhan Jain,
Artur Kuramshin,
Clemens Eppner,
Cyrus Neary,
Edward Hu,
Fabio Ramos,
Jonathan Tremblay,
Kanav Arora,
Kirsty Ellis,
Luca Macesanu,
Matthew Leonard,
Meedeum Cho,
Ozgur Aslan,
Shivin Dass,
Jie Wang,
Xingfang Yuan,
Xuning Yang,
Abhishek Gupta,
Dinesh Jayaraman,
Glen Berseth,
Kostas Daniilidis
, et al. (5 additional authors not shown)
Abstract:
Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environ…
▽ More
Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies.
△ Less
Submitted 22 June, 2025;
originally announced June 2025.
-
Leveraging deep learning for plant disease identification: a bibliometric analysis in SCOPUS from 2018 to 2024
Authors:
Enow Takang Achuo Albert,
Ngalle Hermine Bille,
Ngonkeu Mangaptche Eddy Leonard
Abstract:
This work aimed to present a bibliometric analysis of deep learning research for plant disease identification, with a special focus on generative modeling. A thorough analysis of SCOPUS-sourced bibliometric data from 253 documents was performed. Key performance metrics such as accuracy, precision, recall, and F1-score were analyzed for generative modeling. The findings highlighted significant cont…
▽ More
This work aimed to present a bibliometric analysis of deep learning research for plant disease identification, with a special focus on generative modeling. A thorough analysis of SCOPUS-sourced bibliometric data from 253 documents was performed. Key performance metrics such as accuracy, precision, recall, and F1-score were analyzed for generative modeling. The findings highlighted significant contributions from some authors Too and Arnal Barbedo, whose works had notable citation counts, suggesting their influence on the academic community. Co-authorship networks revealed strong collaborative clusters, while keyword analysis identified emerging research gaps. This study highlights the role of collaboration and citation metrics in shaping research directions and enhancing the impact of scholarly work in applications of deep learning to plant disease identification. Future research should explore the methodologies of highly cited studies to inform best practices and policy-making.
△ Less
Submitted 9 April, 2025;
originally announced April 2025.
-
Deep Neural Imputation: A Framework for Recovering Incomplete Brain Recordings
Authors:
Sabera Talukder,
Jennifer J. Sun,
Matthew Leonard,
Bingni W. Brunton,
Yisong Yue
Abstract:
Neuroscientists and neuroengineers have long relied on multielectrode neural recordings to study the brain. However, in a typical experiment, many factors corrupt neural recordings from individual electrodes, including electrical noise, movement artifacts, and faulty manufacturing. Currently, common practice is to discard these corrupted recordings, reducing already limited data that is difficult…
▽ More
Neuroscientists and neuroengineers have long relied on multielectrode neural recordings to study the brain. However, in a typical experiment, many factors corrupt neural recordings from individual electrodes, including electrical noise, movement artifacts, and faulty manufacturing. Currently, common practice is to discard these corrupted recordings, reducing already limited data that is difficult to collect. To address this challenge, we propose Deep Neural Imputation (DNI), a framework to recover missing values from electrodes by learning from data collected across spatial locations, days, and participants. We explore our framework with a linear nearest-neighbor approach and two deep generative autoencoders, demonstrating DNI's flexibility. One deep autoencoder models participants individually, while the other extends this architecture to model many participants jointly. We evaluate our models across 12 human participants implanted with multielectrode intracranial electrocorticography arrays; participants had no explicit task and behaved naturally across hundreds of recording hours. We show that DNI recovers not only time series but also frequency content, and further establish DNI's practical value by recovering significant performance on a scientifically-relevant downstream neural decoding task.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Atlas-based automated detection of swim bladder in Medaka embryo
Authors:
Diane Genest,
Marc Léonard,
Jean Cousty,
Noémie De Crozé,
Hugues Talbot
Abstract:
Fish embryo models are increasingly being used both for the assessment of chemicals efficacy and potential toxicity. This article proposes a methodology to automatically detect the swim bladder on 2D images of Medaka fish embryos seen either in dorsal view or in lateral view. After embryo segmentation and for each studied orientation, the method builds an atlas of a healthy embryo. This atlas is t…
▽ More
Fish embryo models are increasingly being used both for the assessment of chemicals efficacy and potential toxicity. This article proposes a methodology to automatically detect the swim bladder on 2D images of Medaka fish embryos seen either in dorsal view or in lateral view. After embryo segmentation and for each studied orientation, the method builds an atlas of a healthy embryo. This atlas is then used to define the region of interest and to guide the swim bladder segmentation with a discrete globally optimal active contour. Descriptors are subsequently designed from this segmentation. An automated random forest clas-sifier is built from these descriptors in order to classify embryos with and without a swim bladder. The proposed method is assessed on a dataset of 261 images, containing 202 embryos with a swim bladder (where 196 are in dorsal view and 6 are in lateral view) and 59 without (where 43 are in dorsal view and 16 are in lateral view). We obtain an average precision rate of 95% in the total dataset following 5-fold cross-validation.
△ Less
Submitted 16 February, 2019;
originally announced February 2019.
-
Robust Sequential Detection in Distributed Sensor Networks
Authors:
Mark R. Leonard,
Abdelhak M. Zoubir
Abstract:
We consider the problem of sequential binary hypothesis testing with a distributed sensor network in a non-Gaussian noise environment. To this end, we present a general formulation of the Consensus + Innovations Sequential Probability Ratio Test (CISPRT). Furthermore, we introduce two different concepts for robustifying the CISPRT and propose four different algorithms, namely, the Least-Favorable-…
▽ More
We consider the problem of sequential binary hypothesis testing with a distributed sensor network in a non-Gaussian noise environment. To this end, we present a general formulation of the Consensus + Innovations Sequential Probability Ratio Test (CISPRT). Furthermore, we introduce two different concepts for robustifying the CISPRT and propose four different algorithms, namely, the Least-Favorable-Density-CISPRT, the Median-CISPRT, the M-CISPRT, and the Myriad-CISPRT. Subsequently, we analyze their suitability for different binary hypothesis tests before verifying and evaluating their performance in a shift-in-mean and a shift-in-variance scenario.
△ Less
Submitted 1 February, 2018;
originally announced February 2018.
-
Efficient pattern matching in degenerate strings with the Burrows-Wheeler transform
Authors:
Jacqueline W. Daykin,
Richard Groult,
Yannick Guesnet,
Thierry Lecroq,
Arnaud Lefebvre,
Martine Léonard,
Laurent Mouchard,
Élise Prieur-Gaston,
Bruce Watson
Abstract:
A degenerate or indeterminate string on an alphabet $Σ$ is a sequence of non-empty subsets of $Σ$. Given a degenerate string $t$ of length $n$, we present a new method based on the Burrows--Wheeler transform for searching for a degenerate pattern of length $m$ in $t$ running in $O(mn)$ time on a constant size alphabet $Σ$. Furthermore, it is a hybrid pattern-matching technique that works on both r…
▽ More
A degenerate or indeterminate string on an alphabet $Σ$ is a sequence of non-empty subsets of $Σ$. Given a degenerate string $t$ of length $n$, we present a new method based on the Burrows--Wheeler transform for searching for a degenerate pattern of length $m$ in $t$ running in $O(mn)$ time on a constant size alphabet $Σ$. Furthermore, it is a hybrid pattern-matching technique that works on both regular and degenerate strings. A degenerate string is said to be conservative if its number of non-solid letters is upper-bounded by a fixed positive constant $q$; in this case we show that the search complexity time is $O(qm^2)$. Experimental results show that our method performs well in practice.
△ Less
Submitted 3 August, 2017;
originally announced August 2017.
-
Modelling collaborative services: The COSEMO model
Authors:
Thanh Thoa Pham Thi,
Thang Le Dinh,
Markus Helfert,
Michel Leonard
Abstract:
Despite the dominance of the service sector in the last decades, there is still a need for a strong foundation on service design and innovation. Little attention has paid on service modelling, particularly in the collaboration context. Collaboration is considered as one of solutions for surviving or sustaining the business in the high competitive atmosphere. Collaborative services require various…
▽ More
Despite the dominance of the service sector in the last decades, there is still a need for a strong foundation on service design and innovation. Little attention has paid on service modelling, particularly in the collaboration context. Collaboration is considered as one of solutions for surviving or sustaining the business in the high competitive atmosphere. Collaborative services require various service providers working together according to agreements between them, along with service consumers, in order to co-produce services. In this paper, we address crucial issues in collaborative services such as collaboration levels, sharing data and processes due to business inter-dependencies between service stakeholders. Afterward, we propose a model for Collaborative Service Modelling, which is able to cover identified issues. We also apply our proposed model to modelling an example of healthcare services in order to illustrate the relevance of our modelling approach to the matter in hand.
△ Less
Submitted 11 April, 2017;
originally announced April 2017.
-
FM-index of Alignment with Gaps
Authors:
Joong Chae Na,
Hyunjoon Kim,
Seunghwan Min,
Heejin Park,
Thierry Lecroq,
Martine Leonard,
Laurent Mouchardd,
Kunsoo Park
Abstract:
Recently, a compressed index for similar strings, called the FM-index of alignment (FMA), has been proposed with the functionalities of pattern search and random access. The FMA is quite efficient in space requirement and pattern search time, but it is applicable only for an alignment of similar strings without gaps. In this paper we propose the FM-index of alignment with gaps, a realistic index f…
▽ More
Recently, a compressed index for similar strings, called the FM-index of alignment (FMA), has been proposed with the functionalities of pattern search and random access. The FMA is quite efficient in space requirement and pattern search time, but it is applicable only for an alignment of similar strings without gaps. In this paper we propose the FM-index of alignment with gaps, a realistic index for similar strings, which allows gaps in their alignment. For this, we design a new version of the suffix array of alignment by using alignment transformation and a new definition of the alignment-suffix. The new suffix array of alignment enables us to support the LF-mapping and backward search, the key functionalities of the FM-index, regardless of gap existence in the alignment. We experimentally compared our index with RLCSA due to Makinen et al. on 100 genome sequences from the 1000 Genomes Project. The index size of our index is less than one third of that of RLCSA.
△ Less
Submitted 13 June, 2016;
originally announced June 2016.
-
Multi-Target Tracking in Distributed Sensor Networks using Particle PHD Filters
Authors:
Mark R. Leonard,
Abdelhak M. Zoubir
Abstract:
Multi-target tracking is an important problem in civilian and military applications. This paper investigates multi-target tracking in distributed sensor networks. Data association, which arises particularly in multi-object scenarios, can be tackled by various solutions. We consider sequential Monte Carlo implementations of the Probability Hypothesis Density (PHD) filter based on random finite sets…
▽ More
Multi-target tracking is an important problem in civilian and military applications. This paper investigates multi-target tracking in distributed sensor networks. Data association, which arises particularly in multi-object scenarios, can be tackled by various solutions. We consider sequential Monte Carlo implementations of the Probability Hypothesis Density (PHD) filter based on random finite sets. This approach circumvents the data association issue by jointly estimating all targets in the region of interest. To this end, we develop the Diffusion Particle PHD Filter (D-PPHDF) as well as a centralized version, called the Multi-Sensor Particle PHD Filter (MS-PPHDF). Their performance is evaluated in terms of the Optimal Subpattern Assignment (OSPA) metric, benchmarked against a distributed extension of the Posterior Cramér-Rao Lower Bound (PCRLB), and compared to the performance of an existing distributed PHD Particle Filter. Furthermore, the robustness of the proposed tracking algorithms against outliers and their performance with respect to different amounts of clutter is investigated.
△ Less
Submitted 2 December, 2018; v1 submitted 7 May, 2015;
originally announced May 2015.