Showing 1–22 of 22 results for author: Aghaei, M

Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation

Authors: Max Qiushi Lin, Jincheng Mei, Matin Aghaei, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schuurmans, Csaba Szepesvari, Sharan Vaswani

Abstract: Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with… ▽ More Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as $\texttt{Lin-SPG}$) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of $\texttt{Lin-SPG}$. Under these feature conditions, we prove that $T$ iterations of $\texttt{Lin-SPG}$ with a problem-specific learning rate result in an $O(1/T)$ convergence to the optimal policy. Furthermore, we prove that $\texttt{Lin-SPG}$ with any arbitrary constant learning rate can ensure asymptotic global convergence to the optimal policy. △ Less

Submitted 6 May, 2025; originally announced May 2025.

Comments: 75 pages

Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation

Authors: Lingfeng Zhang, Yuecheng Liu, Zhanguang Zhang, Matin Aghaei, Yaochen Hu, Hongjian Gu, Mohammad Ali Alomrani, David Gamaliel Arcos Bravo, Raika Karimi, Atia Hamidizadeh, Haoping Xu, Guowei Huang, Zhanpeng Zhang, Tongtong Cao, Weichao Qiu, Xingyue Quan, Jianye Hao, Yuzheng Zhuang, Yingxue Zhang

Abstract: Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have made them powerful tools in embodied navigation, enabling agents to leverage commonsense and spatial reasoning for efficient exploration in unfamiliar environments. Existing LLM-based approaches convert global memory, such as semantic or topological maps, into language descriptions to guide navigation. While… ▽ More Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have made them powerful tools in embodied navigation, enabling agents to leverage commonsense and spatial reasoning for efficient exploration in unfamiliar environments. Existing LLM-based approaches convert global memory, such as semantic or topological maps, into language descriptions to guide navigation. While this improves efficiency and reduces redundant exploration, the loss of geometric information in language-based representations hinders spatial reasoning, especially in intricate environments. To address this, VLM-based approaches directly process ego-centric visual inputs to select optimal directions for exploration. However, relying solely on a first-person perspective makes navigation a partially observed decision-making problem, leading to suboptimal decisions in complex environments. In this paper, we present a novel vision-language model (VLM)-based navigation framework that addresses these challenges by adaptively retrieving task-relevant cues from a global memory module and integrating them with the agent's egocentric observations. By dynamically aligning global contextual information with local perception, our approach enhances spatial reasoning and decision-making in long-horizon tasks. Experimental results demonstrate that the proposed method surpasses previous state-of-the-art approaches in object navigation tasks, providing a more effective and scalable solution for embodied navigation. △ Less

Submitted 10 June, 2025; v1 submitted 19 February, 2025; originally announced February 2025.

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

Authors: Michael Lu, Matin Aghaei, Anant Raj, Sharan Vaswani

Abstract: We consider (stochastic) softmax policy gradient (PG) methods for bandits and tabular Markov decision processes (MDPs). While the PG objective is non-concave, recent research has used the objective's smoothness and gradient domination properties to achieve convergence to an optimal policy. However, these theoretical results require setting the algorithm parameters according to unknown problem-depe… ▽ More We consider (stochastic) softmax policy gradient (PG) methods for bandits and tabular Markov decision processes (MDPs). While the PG objective is non-concave, recent research has used the objective's smoothness and gradient domination properties to achieve convergence to an optimal policy. However, these theoretical results require setting the algorithm parameters according to unknown problem-dependent quantities (e.g. the optimal action or the true reward vector in a bandit problem). To address this issue, we borrow ideas from the optimization literature to design practical, principled PG methods in both the exact and stochastic settings. In the exact setting, we employ an Armijo line-search to set the step-size for softmax PG and demonstrate a linear convergence rate. In the stochastic setting, we utilize exponentially decreasing step-sizes, and characterize the convergence rate of the resulting algorithm. We show that the proposed algorithm offers similar theoretical guarantees as the state-of-the art results, but does not require the knowledge of oracle-like quantities. For the multi-armed bandit setting, our techniques result in a theoretically-principled PG algorithm that does not require explicit exploration, the knowledge of the reward gap, the reward distributions, or the noise. Finally, we empirically compare the proposed methods to PG approaches that require oracle knowledge, and demonstrate competitive performance. △ Less

Submitted 30 September, 2024; v1 submitted 21 May, 2024; originally announced May 2024.

Comments: Accepted at RLC 2024

Exploring the Effectiveness of Dataset Synthesis: An application of Apple Detection in Orchards

Authors: Alexander van Meekeren, Maya Aghaei, Klaas Dijkstra

Abstract: Deep object detection models have achieved notable successes in recent years, but one major obstacle remains: the requirement for a large amount of training data. Obtaining such data is a tedious process and is mainly time consuming, leading to the exploration of new research avenues like synthetic data generation techniques. In this study, we explore the usability of Stable Diffusion 2.1-base for… ▽ More Deep object detection models have achieved notable successes in recent years, but one major obstacle remains: the requirement for a large amount of training data. Obtaining such data is a tedious process and is mainly time consuming, leading to the exploration of new research avenues like synthetic data generation techniques. In this study, we explore the usability of Stable Diffusion 2.1-base for generating synthetic datasets of apple trees for object detection and compare it to a baseline model trained on real-world data. After creating a dataset of realistic apple trees with prompt engineering and utilizing a previously trained Stable Diffusion model, the custom dataset was annotated and evaluated by training a YOLOv5m object detection model to predict apples in a real-world apple detection dataset. YOLOv5m was chosen for its rapid inference time and minimal hardware demands. Results demonstrate that the model trained on generated data is slightly underperforming compared to a baseline model trained on real-world images when evaluated on a set of real-world images. However, these findings remain highly promising, as the average precision difference is only 0.09 and 0.06, respectively. Qualitative results indicate that the model can accurately predict the location of apples, except in cases of heavy shading. These findings illustrate the potential of synthetic data generation techniques as a viable alternative to the collection of extensive training data for object detection models. △ Less

Submitted 20 June, 2023; originally announced June 2023.

The Big Data Myth: Using Diffusion Models for Dataset Generation to Train Deep Detection Models

Authors: Roy Voetman, Maya Aghaei, Klaas Dijkstra

Abstract: Despite the notable accomplishments of deep object detection models, a major challenge that persists is the requirement for extensive amounts of training data. The process of procuring such real-world data is a laborious undertaking, which has prompted researchers to explore new avenues of research, such as synthetic data generation techniques. This study presents a framework for the generation of… ▽ More Despite the notable accomplishments of deep object detection models, a major challenge that persists is the requirement for extensive amounts of training data. The process of procuring such real-world data is a laborious undertaking, which has prompted researchers to explore new avenues of research, such as synthetic data generation techniques. This study presents a framework for the generation of synthetic datasets by fine-tuning pretrained stable diffusion models. The synthetic datasets are then manually annotated and employed for training various object detection models. These detectors are evaluated on a real-world test set of 331 images and compared against a baseline model that was trained on real-world images. The results of this study reveal that the object detection models trained on synthetic data perform similarly to the baseline model. In the context of apple detection in orchards, the average precision deviation with the baseline ranges from 0.09 to 0.12. This study illustrates the potential of synthetic data generation techniques as a viable alternative to the collection of extensive training data for the training of deep models. △ Less

Submitted 16 June, 2023; originally announced June 2023.

Crime scene classification from skeletal trajectory analysis in surveillance settings

Authors: Alina-Daniela Matei, Estefania Talavera, Maya Aghaei

Abstract: Video anomaly analysis is a core task actively pursued in the field of computer vision, with applications extending to real-world crime detection in surveillance footage. In this work, we address the task of human-related crime classification. In our proposed approach, the human body in video frames, represented as skeletal joints trajectories, is used as the main source of exploration. First, we… ▽ More Video anomaly analysis is a core task actively pursued in the field of computer vision, with applications extending to real-world crime detection in surveillance footage. In this work, we address the task of human-related crime classification. In our proposed approach, the human body in video frames, represented as skeletal joints trajectories, is used as the main source of exploration. First, we introduce the significance of extending the ground truth labels for HR-Crime dataset and hence, propose a supervised and unsupervised methodology to generate trajectory-level ground truth labels. Next, given the availability of the trajectory-level ground truth, we introduce a trajectory-based crime classification framework. Ablation studies are conducted with various architectures and feature fusion strategies for the representation of the human trajectories. The conducted experiments demonstrate the feasibility of the task and pave the path for further research in the field. △ Less

Submitted 4 July, 2022; originally announced July 2022.

Hyper-Spectral Imaging for Overlapping Plastic Flakes Segmentation

Authors: Guillem Martinez, Maya Aghaei, Martin Dijkstra, Bhalaji Nagarajan, Femke Jaarsma, Jaap van de Loosdrecht, Petia Radeva, Klaas Dijkstra

Abstract: Given the hyper-spectral imaging unique potentials in grasping the polymer characteristics of different materials, it is commonly used in sorting procedures. In a practical plastic sorting scenario, multiple plastic flakes may overlap which depending on their characteristics, the overlap can be reflected in their spectral signature. In this work, we use hyper-spectral imaging for the segmentation… ▽ More Given the hyper-spectral imaging unique potentials in grasping the polymer characteristics of different materials, it is commonly used in sorting procedures. In a practical plastic sorting scenario, multiple plastic flakes may overlap which depending on their characteristics, the overlap can be reflected in their spectral signature. In this work, we use hyper-spectral imaging for the segmentation of three types of plastic flakes and their possible overlapping combinations. We propose an intuitive and simple multi-label encoding approach, bitfield encoding, to account for the overlapping regions. With our experiments, we show that the bitfield encoding improves over the baseline single-label approach and we further demonstrate its potential in predicting multiple labels for overlapping classes even when the model is only trained with non-overlapping classes. △ Less

Submitted 23 March, 2022; originally announced March 2022.

Comments: Submitted to ICIP2022

On the Effect of Pre-Processing and Model Complexity for Plastic Analysis Using Short-Wave-Infrared Hyper-Spectral Imaging

Authors: Klaas Dijkstra, Maya Aghaei, Femke Jaarsma, Martin Dijkstra, Rudy Folkersma, Jan Jager, Jaap van de Loosdrecht

Abstract: The importance of plastic waste recycling is undeniable. In this respect, computer vision and deep learning enable solutions through the automated analysis of short-wave-infrared hyper-spectral images of plastics. In this paper, we offer an exhaustive empirical study to show the importance of efficient model selection for resolving the task of hyper-spectral image segmentation of various plastic f… ▽ More The importance of plastic waste recycling is undeniable. In this respect, computer vision and deep learning enable solutions through the automated analysis of short-wave-infrared hyper-spectral images of plastics. In this paper, we offer an exhaustive empirical study to show the importance of efficient model selection for resolving the task of hyper-spectral image segmentation of various plastic flakes using deep learning. We assess the complexity level of generic and specialized models and infer their performance capacity: generic models are often unnecessarily complex. We introduce two variants of a specialized hyper-spectral architecture, PlasticNet, that outperforms several well-known segmentation architectures in both performance as well as computational complexity. In addition, we shed lights on the significance of signal pre-processing within the realm of hyper-spectral imaging. To complete our contribution, we introduce the largest, most versatile hyper-spectral dataset of plastic flakes of four primary polymer types. △ Less

Submitted 21 March, 2022; originally announced March 2022.

HR-Crime: Human-Related Anomaly Detection in Surveillance Videos

Authors: Kayleigh Boekhoudt, Alina Matei, Maya Aghaei, Estefanía Talavera

Abstract: The automatic detection of anomalies captured by surveillance settings is essential for speeding the otherwise laborious approach. To date, UCF-Crime is the largest available dataset for automatic visual analysis of anomalies and consists of real-world crime scenes of various categories. In this paper, we introduce HR-Crime, a subset of the UCF-Crime dataset suitable for human-related anomaly dete… ▽ More The automatic detection of anomalies captured by surveillance settings is essential for speeding the otherwise laborious approach. To date, UCF-Crime is the largest available dataset for automatic visual analysis of anomalies and consists of real-world crime scenes of various categories. In this paper, we introduce HR-Crime, a subset of the UCF-Crime dataset suitable for human-related anomaly detection tasks. We rely on state-of-the-art techniques to build the feature extraction pipeline for human-related anomaly detection. Furthermore, we present the baseline anomaly detection analysis on the HR-Crime. HR-Crime as well as the developed feature extraction pipeline and the extracted features will be publicly available for further research in the field. △ Less

Submitted 31 July, 2021; originally announced August 2021.

Comments: Accepted by CAIP 2021

Obstacle Detection for BVLOS Drones

Authors: Jan Moros Esteban, Jaap van de Loosdrecht, Maya Aghaei

Abstract: With the introduction of new regulations in the European Union, the future of Beyond Visual Line Of Sight (BVLOS) drones is set to bloom. This led to the creation of the theBEAST project, which aims to create an autonomous security drone, with focus on those regulations and on safety. This technical paper describes the first steps of a module within this project, which revolves around detecting ob… ▽ More With the introduction of new regulations in the European Union, the future of Beyond Visual Line Of Sight (BVLOS) drones is set to bloom. This led to the creation of the theBEAST project, which aims to create an autonomous security drone, with focus on those regulations and on safety. This technical paper describes the first steps of a module within this project, which revolves around detecting obstacles so they can be avoided in a fail-safe landing. A deep learning powered object detection method is the subject of our research, and various experiments are held to maximize its performance, such as comparing various data augmentation techniques or YOLOv3 and YOLOv5. According to the results of the experiments, we conclude that although object detection is a promising approach to resolve this problem, more volume of data is required for potential usage in a real-life application. △ Less

Submitted 22 June, 2021; v1 submitted 21 June, 2021; originally announced June 2021.

Comments: 7 pages, 7 figures, Supervisors: Maya Aghaei Gavari and Jaap van de Loosdrecht

Single Image Human Proxemics Estimation for Visual Social Distancing

Authors: Maya Aghaei, Matteo Bustreo, Yiming Wang, Gianluca Bailo, Pietro Morerio, Alessio Del Bue

Abstract: In this work, we address the problem of estimating the so-called "Social Distancing" given a single uncalibrated image in unconstrained scenarios. Our approach proposes a semi-automatic solution to approximate the homography matrix between the scene ground and image plane. With the estimated homography, we then leverage an off-the-shelf pose detector to detect body poses on the image and to reason… ▽ More In this work, we address the problem of estimating the so-called "Social Distancing" given a single uncalibrated image in unconstrained scenarios. Our approach proposes a semi-automatic solution to approximate the homography matrix between the scene ground and image plane. With the estimated homography, we then leverage an off-the-shelf pose detector to detect body poses on the image and to reason upon their inter-personal distances using the length of their body-parts. Inter-personal distances are further locally inspected to detect possible violations of the social distancing rules. We validate our proposed method quantitatively and qualitatively against baselines on public domain datasets for which we provided groundtruth on inter-personal distances. Besides, we demonstrate the application of our method deployed in a real testing scenario where statistics on the inter-personal distances are currently used to improve the safety in a critical environment. △ Less

Submitted 5 November, 2020; v1 submitted 3 November, 2020; originally announced November 2020.

Comments: Paper accepted at WACV 2021 conference

Complex-Object Visual Inspection via Multiple Lighting Configurations

Authors: Maya Aghaei, Matteo Bustreo, Pietro Morerio, Nicolo Carissimi, Alessio Del Bue, Vittorio Murino

Abstract: The design of an automatic visual inspection system is usually performed in two stages. While the first stage consists in selecting the most suitable hardware setup for highlighting most effectively the defects on the surface to be inspected, the second stage concerns the development of algorithmic solutions to exploit the potentials offered by the collected data. In this paper, first, we presen… ▽ More The design of an automatic visual inspection system is usually performed in two stages. While the first stage consists in selecting the most suitable hardware setup for highlighting most effectively the defects on the surface to be inspected, the second stage concerns the development of algorithmic solutions to exploit the potentials offered by the collected data. In this paper, first, we present a novel illumination setup embedding four illumination configurations to resemble diffused, dark-field, and front lighting techniques. Second, we analyze the contributions brought by deploying the proposed setup in training phase only - mimicking the scenario in which an already developed visual inspection system cannot be modified on the customer site - and in evaluation phase. Along with an exhaustive set of experiments, in this paper, we demonstrate the suitability of the proposed setup for effective illumination of complex-objects, defined as manufactured items with variable surface characteristics that cannot be determined a priori. Moreover, we discuss the importance of multiple light configurations availability during training and their natural boosting effect which, without the need to modify the system design in evaluation phase, lead to improvements in the overall system performance. △ Less

Submitted 20 April, 2020; originally announced April 2020.

Comments: 8 pages, 7 figures, submitted to ICPR2020

Understanding Deep Architectures by Visual Summaries

Authors: Marco Carletti, Marco Godi, Maedeh Aghaei, Francesco Giuliari, Marco Cristani

Abstract: In deep learning, visualization techniques extract the salient patterns exploited by deep networks for image classification, focusing on single images; no effort has been spent in investigating whether these patterns are systematically related to precise semantic entities over multiple images belonging to a same class, thus failing to capture the very understanding of the image class the network h… ▽ More In deep learning, visualization techniques extract the salient patterns exploited by deep networks for image classification, focusing on single images; no effort has been spent in investigating whether these patterns are systematically related to precise semantic entities over multiple images belonging to a same class, thus failing to capture the very understanding of the image class the network has realized. This paper goes in this direction, presenting a visualization framework which produces a group of clusters or summaries, each one formed by crisp salient image regions focusing on a particular part that the network has exploited with high regularity to decide for a given class. The approach is based on a sparse optimization step providing sharp image saliency masks that are clustered together by means of a semantic flow similarity measure. The summaries communicate clearly what a network has exploited of a particular image class, and this is proved through automatic image tagging and with a user study. Beyond the deep network understanding, summaries are also useful for many quantitative reasons: their number is correlated with ability of a network to classify (more summaries, better performances), and they can be used to improve the classification accuracy of a network through summary-driven specializations. △ Less

Submitted 29 August, 2019; v1 submitted 27 January, 2018; originally announced January 2018.

Comments: Project page and code available at http://marcocarletti.altervista.org/publications/understanding-visual-summaries/

Social Style Characterization from Egocentric Photo-streams

Authors: Maedeh Aghaei, Mariella Dimiccoli, Cristian Canton Ferrer, Petia Radeva

Abstract: This paper proposes a system for automatic social pattern characterization using a wearable photo-camera. The proposed pipeline consists of three major steps. First, detection of people with whom the camera wearer interacts and, second, categorization of the detected social interactions into formal and informal. These two steps act at event-level where each potential social event is modeled as a m… ▽ More This paper proposes a system for automatic social pattern characterization using a wearable photo-camera. The proposed pipeline consists of three major steps. First, detection of people with whom the camera wearer interacts and, second, categorization of the detected social interactions into formal and informal. These two steps act at event-level where each potential social event is modeled as a multi-dimensional time-series, whose dimensions correspond to a set of relevant features for each task, and a LSTM network is employed for time-series classification. In the last step, recurrences of the same person across the whole set of social interactions are clustered to achieve a comprehensive understanding of the diversity and frequency of the social relations of the user. Experiments over a dataset acquired by a user wearing a photo-camera during a month show promising results on the task of social pattern characterization from egocentric photo-streams. △ Less

Submitted 18 September, 2017; originally announced September 2017.

Comments: International Conference on Computer Vision (ICCV). Workshop on Egocentric Percetion, Interaction and Computing

Towards social pattern characterization in egocentric photo-streams

Authors: Maedeh Aghaei, Mariella Dimiccoli, Cristian Canton Ferrer, Petia Radeva

Abstract: Following the increasingly popular trend of social interaction analysis in egocentric vision, this manuscript presents a comprehensive study for automatic social pattern characterization of a wearable photo-camera user, by relying on the visual analysis of egocentric photo-streams. The proposed framework consists of three major steps. The first step is to detect social interactions of the user whe… ▽ More Following the increasingly popular trend of social interaction analysis in egocentric vision, this manuscript presents a comprehensive study for automatic social pattern characterization of a wearable photo-camera user, by relying on the visual analysis of egocentric photo-streams. The proposed framework consists of three major steps. The first step is to detect social interactions of the user where the impact of several social signals on the task is explored. The detected social events are inspected in the second step for categorization into different social meetings. These two steps act at event-level where each potential social event is modeled as a multi-dimensional time-series, whose dimensions correspond to a set of relevant features for each task, and LSTM is employed to classify the time-series. The last step of the framework is to characterize social patterns, which is essentially to infer the diversity and frequency of the social relations of the user through discovery of recurrences of the same people across the whole set of social events of the user. Experimental evaluation over a dataset acquired by 9 users demonstrates promising results on the task of social pattern characterization from egocentric photo-streams. △ Less

Submitted 9 January, 2018; v1 submitted 5 September, 2017; originally announced September 2017.

Comments: 42 pages, 14 figures. Submitted to Elsevier, Computer Vision and Image Understanding (Under Review)

R-Clustering for Egocentric Video Segmentation

Authors: Estefania Talavera, Mariella Dimiccoli, Marc Bolaños, Maedeh Aghaei, Petia Radeva

Abstract: In this paper, we present a new method for egocentric video temporal segmentation based on integrating a statistical mean change detector and agglomerative clustering(AC) within an energy-minimization framework. Given the tendency of most AC methods to oversegment video sequences when clustering their frames, we combine the clustering with a concept drift detection technique (ADWIN) that has rigor… ▽ More In this paper, we present a new method for egocentric video temporal segmentation based on integrating a statistical mean change detector and agglomerative clustering(AC) within an energy-minimization framework. Given the tendency of most AC methods to oversegment video sequences when clustering their frames, we combine the clustering with a concept drift detection technique (ADWIN) that has rigorous guarantee of performances. ADWIN serves as a statistical upper bound for the clustering-based video segmentation. We integrate both techniques in an energy-minimization framework that serves to disambiguate the decision of both techniques and to complete the segmentation taking into account the temporal continuity of video frames descriptors. We present experiments over egocentric sets of more than 13.000 images acquired with different wearable cameras, showing that our method outperforms state-of-the-art clustering methods. △ Less

Submitted 10 April, 2017; originally announced April 2017.

Clothing and People - A Social Signal Processing Perspective

Authors: Maedeh Aghaei, Federico Parezzan, Mariella Dimiccoli, Petia Radeva, Marco Cristani

Abstract: In our society and century, clothing is not anymore used only as a means for body protection. Our paper builds upon the evidence, studied within the social sciences, that clothing brings a clear communicative message in terms of social signals, influencing the impression and behaviour of others towards a person. In fact, clothing correlates with personality traits, both in terms of self-assessment… ▽ More In our society and century, clothing is not anymore used only as a means for body protection. Our paper builds upon the evidence, studied within the social sciences, that clothing brings a clear communicative message in terms of social signals, influencing the impression and behaviour of others towards a person. In fact, clothing correlates with personality traits, both in terms of self-assessment and assessments that unacquainted people give to an individual. The consequences of these facts are important: the influence of clothing on the decision making of individuals has been investigated in the literature, showing that it represents a discriminative factor to differentiate among diverse groups of people. Unfortunately, this has been observed after cumbersome and expensive manual annotations, on very restricted populations, limiting the scope of the resulting claims. With this position paper, we want to sketch the main steps of the very first systematic analysis, driven by social signal processing techniques, of the relationship between clothing and social signals, both sent and perceived. Thanks to human parsing technologies, which exhibit high robustness owing to deep learning architectures, we are now capable to isolate visual patterns characterising a large types of garments. These algorithms will be used to capture statistical relations on a large corpus of evidence to confirm the sociological findings and to go beyond the state of the art. △ Less

Submitted 7 April, 2017; originally announced April 2017.

Comments: To appear in the 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017)

All the people around me: face discovery in egocentric photo-streams

Authors: Maedeh Aghaei, Mariella Dimiccoli, Petia Radeva

Abstract: Given an unconstrained stream of images captured by a wearable photo-camera (2fpm), we propose an unsupervised bottom-up approach for automatic clustering appearing faces into the individual identities present in these data. The problem is challenging since images are acquired under real world conditions; hence the visible appearance of the people in the images undergoes intensive variations. Our… ▽ More Given an unconstrained stream of images captured by a wearable photo-camera (2fpm), we propose an unsupervised bottom-up approach for automatic clustering appearing faces into the individual identities present in these data. The problem is challenging since images are acquired under real world conditions; hence the visible appearance of the people in the images undergoes intensive variations. Our proposed pipeline consists of first arranging the photo-stream into events, later, localizing the appearance of multiple people in them, and finally, grouping various appearances of the same person across different events. Experimental results performed on a dataset acquired by wearing a photo-camera during one month, demonstrate the effectiveness of the proposed approach for the considered purpose. △ Less

Submitted 12 May, 2017; v1 submitted 6 March, 2017; originally announced March 2017.

Comments: 5 pages, 3 figures, accepted in IEEE International Conference on Image Processing (ICIP 2017)

Rejecting inadmissible rules in reduced normal forms in S4

Authors: Mojtaba Aghaei, Maryam Rostami Giv

Abstract: Several methods for checking admissibility of rules in the modal logic $S4$ are presented in [1], [15]. These methods determine admissibility of rules in $S4$, but they don't determine or give substitutions rejecting inadmissible rules. In this paper, we investigate some relations between one of the above methods, based on the reduced normal form rules, and sets of substitutions which reject them.… ▽ More Several methods for checking admissibility of rules in the modal logic $S4$ are presented in [1], [15]. These methods determine admissibility of rules in $S4$, but they don't determine or give substitutions rejecting inadmissible rules. In this paper, we investigate some relations between one of the above methods, based on the reduced normal form rules, and sets of substitutions which reject them. We also generalize the method in [1], [15] for one rule to admissibility of a set of rules. △ Less

Submitted 18 January, 2017; originally announced January 2017.

MSC Class: 03B47; 03D15

With Whom Do I Interact? Detecting Social Interactions in Egocentric Photo-streams

Authors: Maedeh Aghaei, Mariella Dimiccoli, Petia Radeva

Abstract: Given a user wearing a low frame rate wearable camera during a day, this work aims to automatically detect the moments when the user gets engaged into a social interaction solely by reviewing the automatically captured photos by the worn camera. The proposed method, inspired by the sociological concept of F-formation, exploits distance and orientation of the appearing individuals -with respect to… ▽ More Given a user wearing a low frame rate wearable camera during a day, this work aims to automatically detect the moments when the user gets engaged into a social interaction solely by reviewing the automatically captured photos by the worn camera. The proposed method, inspired by the sociological concept of F-formation, exploits distance and orientation of the appearing individuals -with respect to the user- in the scene from a bird-view perspective. As a result, the interaction pattern over the sequence can be understood as a two-dimensional time series that corresponds to the temporal evolution of the distance and orientation features over time. A Long-Short Term Memory-based Recurrent Neural Network is then trained to classify each time series. Experimental evaluation over a dataset of 30.000 images has shown promising results on the proposed method for social interaction detection in egocentric photo-streams. △ Less

Submitted 12 May, 2017; v1 submitted 13 May, 2016; originally announced May 2016.

Comments: 6 pages, 9 figures, accepted and presented in International Conference on Pattern Recognition (ICPR 2016)

SR-Clustering: Semantic Regularized Clustering for Egocentric Photo Streams Segmentation

Authors: Mariella Dimiccoli, Marc Bolaños, Estefania Talavera, Maedeh Aghaei, Stavri G. Nikolov, Petia Radeva

Abstract: While wearable cameras are becoming increasingly popular, locating relevant information in large unstructured collections of egocentric images is still a tedious and time consuming processes. This paper addresses the problem of organizing egocentric photo streams acquired by a wearable camera into semantically meaningful segments. First, contextual and semantic information is extracted for each im… ▽ More While wearable cameras are becoming increasingly popular, locating relevant information in large unstructured collections of egocentric images is still a tedious and time consuming processes. This paper addresses the problem of organizing egocentric photo streams acquired by a wearable camera into semantically meaningful segments. First, contextual and semantic information is extracted for each image by employing a Convolutional Neural Networks approach. Later, by integrating language processing, a vocabulary of concepts is defined in a semantic space. Finally, by exploiting the temporal coherence in photo streams, images which share contextual and semantic attributes are grouped together. The resulting temporal segmentation is particularly suited for further analysis, ranging from activity and event recognition to semantic indexing and summarization. Experiments over egocentric sets of nearly 17,000 images, show that the proposed approach outperforms state-of-the-art methods. △ Less

Submitted 17 October, 2016; v1 submitted 22 December, 2015; originally announced December 2015.

Comments: 23 pages, 10 figures, 2 tables. In Press in Computer Vision and Image Understanding Journal

Multi-Face Tracking by Extended Bag-of-Tracklets in Egocentric Videos

Authors: Maedeh Aghaei, Mariella Dimiccoli, Petia Radeva

Abstract: Wearable cameras offer a hands-free way to record egocentric images of daily experiences, where social events are of special interest. The first step towards detection of social events is to track the appearance of multiple persons involved in it. In this paper, we propose a novel method to find correspondences of multiple faces in low temporal resolution egocentric videos acquired through a weara… ▽ More Wearable cameras offer a hands-free way to record egocentric images of daily experiences, where social events are of special interest. The first step towards detection of social events is to track the appearance of multiple persons involved in it. In this paper, we propose a novel method to find correspondences of multiple faces in low temporal resolution egocentric videos acquired through a wearable camera. This kind of photo-stream imposes additional challenges to the multi-tracking problem with respect to conventional videos. Due to the free motion of the camera and to its low temporal resolution, abrupt changes in the field of view, in illumination condition and in the target location are highly frequent. To overcome such difficulties, we propose a multi-face tracking method that generates a set of tracklets through finding correspondences along the whole sequence for each detected face and takes advantage of the tracklets redundancy to deal with unreliable ones. Similar tracklets are grouped into the so called extended bag-of-tracklets (eBoT), which is aimed to correspond to a specific person. Finally, a prototype tracklet is extracted for each eBoT, where the occurred occlusions are estimated by relying on a new measure of confidence. We validated our approach over an extensive dataset of egocentric photo-streams and compared it to state of the art methods, demonstrating its effectiveness and robustness. △ Less

Submitted 13 January, 2016; v1 submitted 16 July, 2015; originally announced July 2015.

Comments: 27 pages, 18 figures, submitted to computer vision and image understanding journal

Report number: YCVIU2393