-
Random Forest Calibration
Authors:
Mohammad Hossein Shaker,
Eyke Hüllermeier
Abstract:
The Random Forest (RF) classifier is often claimed to be relatively well calibrated when compared with other machine learning methods. Moreover, the existing literature suggests that traditional calibration methods, such as isotonic regression, do not substantially enhance the calibration of RF probability estimates unless supplied with extensive calibration data sets, which can represent a signif…
▽ More
The Random Forest (RF) classifier is often claimed to be relatively well calibrated when compared with other machine learning methods. Moreover, the existing literature suggests that traditional calibration methods, such as isotonic regression, do not substantially enhance the calibration of RF probability estimates unless supplied with extensive calibration data sets, which can represent a significant obstacle in cases of limited data availability. Nevertheless, there seems to be no comprehensive study validating such claims and systematically comparing state-of-the-art calibration methods specifically for RF. To close this gap, we investigate a broad spectrum of calibration methods tailored to or at least applicable to RF, ranging from scaling techniques to more advanced algorithms. Our results based on synthetic as well as real-world data unravel the intricacies of RF probability estimates, scrutinize the impacts of hyper-parameters, compare calibration methods in a systematic way. We show that a well-optimized RF performs as well as or better than leading calibration approaches.
△ Less
Submitted 28 January, 2025;
originally announced January 2025.
-
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
Authors:
Ashmal Vayani,
Dinura Dissanayake,
Hasindri Watawana,
Noor Ahsan,
Nevasini Sasikumar,
Omkar Thawakar,
Henok Biadglign Ademtew,
Yahya Hmaiti,
Amandeep Kumar,
Kartik Kuckreja,
Mykola Maslych,
Wafa Al Ghallabi,
Mihail Mihaylov,
Chao Qin,
Abdelrahman M Shaker,
Mike Zhang,
Mahardika Krisna Ihsani,
Amiel Esplana,
Monil Gokani,
Shachar Mirkin,
Harsh Singh,
Ashay Srivastava,
Endre Hamerlik,
Fathinah Asma Izzati,
Fadillah Adamsyah Maani
, et al. (44 additional authors not shown)
Abstract:
Existing Large Multimodal Models (LMMs) generally focus on only a few regions and languages. As LMMs continue to improve, it is increasingly important to ensure they understand cultural contexts, respect local sensitivities, and support low-resource languages, all while effectively integrating corresponding visual cues. In pursuit of culturally diverse global multimodal models, our proposed All La…
▽ More
Existing Large Multimodal Models (LMMs) generally focus on only a few regions and languages. As LMMs continue to improve, it is increasingly important to ensure they understand cultural contexts, respect local sensitivities, and support low-resource languages, all while effectively integrating corresponding visual cues. In pursuit of culturally diverse global multimodal models, our proposed All Languages Matter Benchmark (ALM-bench) represents the largest and most comprehensive effort to date for evaluating LMMs across 100 languages. ALM-bench challenges existing models by testing their ability to understand and reason about culturally diverse images paired with text in various languages, including many low-resource languages traditionally underrepresented in LMM research. The benchmark offers a robust and nuanced evaluation framework featuring various question formats, including true/false, multiple choice, and open-ended questions, which are further divided into short and long-answer categories. ALM-bench design ensures a comprehensive assessment of a model's ability to handle varied levels of difficulty in visual and linguistic reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully curates content from 13 distinct cultural aspects, ranging from traditions and rituals to famous personalities and celebrations. Through this, ALM-bench not only provides a rigorous testing ground for state-of-the-art open and closed-source LMMs but also highlights the importance of cultural and linguistic inclusivity, encouraging the development of models that can serve diverse global populations effectively. Our benchmark is publicly available.
△ Less
Submitted 30 April, 2025; v1 submitted 25 November, 2024;
originally announced November 2024.
-
Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference
Authors:
Mohammad Hossein Shaker,
Eyke Hüllermeier
Abstract:
The idea to distinguish and quantify two important types of uncertainty, often referred to as aleatoric and epistemic, has received increasing attention in machine learning research in the last couple of years. In this paper, we consider ensemble-based approaches to uncertainty quantification. Distinguishing between different types of uncertainty-aware learning algorithms, we specifically focus on…
▽ More
The idea to distinguish and quantify two important types of uncertainty, often referred to as aleatoric and epistemic, has received increasing attention in machine learning research in the last couple of years. In this paper, we consider ensemble-based approaches to uncertainty quantification. Distinguishing between different types of uncertainty-aware learning algorithms, we specifically focus on Bayesian methods and approaches based on so-called credal sets, which naturally suggest themselves from an ensemble learning point of view. For both approaches, we address the question of how to quantify aleatoric and epistemic uncertainty. The effectiveness of corresponding measures is evaluated and compared in an empirical study on classification with a reject option.
△ Less
Submitted 10 December, 2021; v1 submitted 21 July, 2021;
originally announced July 2021.
-
Online rating system development using blockchain-based distributed ledger technology
Authors:
Monir Shaker,
Fereidoon Shams Aliee,
Reza Fotohi
Abstract:
In most websites, the online rating system provides the ratings of products and services to users. Lack of trust in data integrity and its manipulation has hindered fulfilling user satisfaction. Since existing online rating systems deal with a central server, all rating data is stored on the central server. Therefore, all rating data can be removed, modified, and manipulated by the system manager…
▽ More
In most websites, the online rating system provides the ratings of products and services to users. Lack of trust in data integrity and its manipulation has hindered fulfilling user satisfaction. Since existing online rating systems deal with a central server, all rating data is stored on the central server. Therefore, all rating data can be removed, modified, and manipulated by the system manager to change the ratings in favor of the service or product provider. In this paper, an online rating system using distributed ledger technologies has been presented as the proposed system to solve all the weaknesses of current systems. Distributed ledger technologies are completely decentralized and there is no centralization on them by any institution. Distributed ledger technologies have different variants. Among distributed ledger technologies, blockchain technology has been used in the proposed rating system because of its support for smart contracts. In the proposed online rating system, the Ethereum platform has been chosen from different blockchain platforms that have a public permission network. In this system, the raters cannot rate unless they submit a request to the system and be authorized to take part in the online product rating process. The important feature of the Ethereum platform is its support for smart contracts, which can be used to write the rating contract in the Solidity language. Also, using Proof of Authority consensus mechanisms, all rating transactions are approved by the surveyors. Since in the real Ethereum system, each rating transaction is sent to the network by the raters, some gas must be paid for each rating transaction. However, since this method is expensive, TestNet blockchain can be used in the rating system. Finally, the proposed rating system was used for rating the restaurants of a website and its features were tested.
△ Less
Submitted 11 January, 2021;
originally announced January 2021.
-
Robust Real-Time Pedestrian Detection on Embedded Devices
Authors:
Mohamed Afifi,
Yara Ali,
Karim Amer,
Mahmoud Shaker,
Mohamed Elhelw
Abstract:
Detection of pedestrians on embedded devices, such as those on-board of robots and drones, has many applications including road intersection monitoring, security, crowd monitoring and surveillance, to name a few. However, the problem can be challenging due to continuously-changing camera viewpoint and varying object appearances as well as the need for lightweight algorithms suitable for embedded s…
▽ More
Detection of pedestrians on embedded devices, such as those on-board of robots and drones, has many applications including road intersection monitoring, security, crowd monitoring and surveillance, to name a few. However, the problem can be challenging due to continuously-changing camera viewpoint and varying object appearances as well as the need for lightweight algorithms suitable for embedded systems. This paper proposes a robust framework for pedestrian detection in many footages. The framework performs fine and coarse detections on different image regions and exploits temporal and spatial characteristics to attain enhanced accuracy and real time performance on embedded boards. The framework uses the Yolo-v3 object detection [1] as its backbone detector and runs on the Nvidia Jetson TX2 embedded board, however other detectors and/or boards can be used as well. The performance of the framework is demonstrated on two established datasets and its achievement of the second place in CVPR 2019 Embedded Real-Time Inference (ERTI) Challenge.
△ Less
Submitted 13 December, 2020;
originally announced December 2020.
-
Aleatoric and Epistemic Uncertainty with Random Forests
Authors:
Mohammad Hossein Shaker,
Eyke Hüllermeier
Abstract:
Due to the steadily increasing relevance of machine learning for practical applications, many of which are coming with safety requirements, the notion of uncertainty has received increasing attention in machine learning research in the last couple of years. In particular, the idea of distinguishing between two important types of uncertainty, often refereed to as aleatoric and epistemic, has recent…
▽ More
Due to the steadily increasing relevance of machine learning for practical applications, many of which are coming with safety requirements, the notion of uncertainty has received increasing attention in machine learning research in the last couple of years. In particular, the idea of distinguishing between two important types of uncertainty, often refereed to as aleatoric and epistemic, has recently been studied in the setting of supervised learning. In this paper, we propose to quantify these uncertainties with random forests. More specifically, we show how two general approaches for measuring the learner's aleatoric and epistemic uncertainty in a prediction can be instantiated with decision trees and random forests as learning algorithms in a classification setting. In this regard, we also compare random forests with deep neural networks, which have been used for a similar purpose.
△ Less
Submitted 3 January, 2020;
originally announced January 2020.
-
Robust Real-time Pedestrian Detection in Aerial Imagery on Jetson TX2
Authors:
Mohamed Afifi,
Yara Ali,
Karim Amer,
Mahmoud Shaker,
Mohamed ElHelw
Abstract:
Detection of pedestrians in aerial imagery captured by drones has many applications including intersection monitoring, patrolling, and surveillance, to name a few. However, the problem is involved due to continuouslychanging camera viewpoint and object appearance as well as the need for lightweight algorithms to run on on-board embedded systems. To address this issue, the paper proposes a framewor…
▽ More
Detection of pedestrians in aerial imagery captured by drones has many applications including intersection monitoring, patrolling, and surveillance, to name a few. However, the problem is involved due to continuouslychanging camera viewpoint and object appearance as well as the need for lightweight algorithms to run on on-board embedded systems. To address this issue, the paper proposes a framework for pedestrian detection in videos based on the YOLO object detection network [6] while having a high throughput of more than 5 FPS on the Jetson TX2 embedded board. The framework exploits deep learning for robust operation and uses a pre-trained model without the need for any additional training which makes it flexible to apply on different setups with minimum amount of tuning. The method achieves ~81 mAP when applied on a sample video from the Embedded Real-Time Inference (ERTI) Challenge where pedestrians are monitored by a UAV.
△ Less
Submitted 16 May, 2019;
originally announced May 2019.
-
Drone Path-Following in GPS-Denied Environments using Convolutional Networks
Authors:
M. Samy,
K. Amer,
M. Shaker,
M. ElHelw
Abstract:
his paper presents a simple approach for drone navigation to follow a predetermined path using visual input only without reliance on a Global Positioning System (GPS). A Convolutional Neural Network (CNN) is used to output the steering command of the drone in an end-to-end approach. We tested our approach in two simulated environments in the Unreal Engine using the AirSim plugin for drone simulati…
▽ More
his paper presents a simple approach for drone navigation to follow a predetermined path using visual input only without reliance on a Global Positioning System (GPS). A Convolutional Neural Network (CNN) is used to output the steering command of the drone in an end-to-end approach. We tested our approach in two simulated environments in the Unreal Engine using the AirSim plugin for drone simulation. Results show that the proposed approach, despite its simplicity, has average cross track distance less than 2.9 meters in the simulated environment. We also investigate the significance of data augmentation in path following. Finally, we conclude by suggesting possible enhancements for extending our approach to more difficult paths in real life, in the hope that one day visual navigation will become the norm in GPS-denied zones.
△ Less
Submitted 5 May, 2019;
originally announced May 2019.
-
Deep Convolutional Neural Network-Based Autonomous Drone Navigation
Authors:
K. Amer,
M. Samy,
M. Shaker,
M. ElHelw
Abstract:
This paper presents a novel approach for aerial drone autonomous navigation along predetermined paths using only visual input form an onboard camera and without reliance on a Global Positioning System (GPS). It is based on using a deep Convolutional Neural Network (CNN) combined with a regressor to output the drone steering commands. Furthermore, multiple auxiliary navigation paths that form a nav…
▽ More
This paper presents a novel approach for aerial drone autonomous navigation along predetermined paths using only visual input form an onboard camera and without reliance on a Global Positioning System (GPS). It is based on using a deep Convolutional Neural Network (CNN) combined with a regressor to output the drone steering commands. Furthermore, multiple auxiliary navigation paths that form a navigation envelope are used for data augmentation to make the system adaptable to real-life deployment scenarios. The approach is suitable for automating drone navigation in applications that exhibit regular trips or visits to same locations such as environmental and desertification monitoring, parcel/aid delivery and drone-based wireless internet delivery. In this case, the proposed algorithm replaces human operators, enhances accuracy of GPS-based map navigation, alleviates problems related to GPS-spoofing and enables navigation in GPS-denied environments. Our system is tested in two scenarios using the Unreal Engine-based AirSim plugin for drone simulation with promising results of average cross track distance less than 1.4 meters and mean waypoints minimum distance of less than 1 meter.
△ Less
Submitted 5 May, 2019;
originally announced May 2019.
-
Deep Reinforcement Learning for Dexterous Manipulation with Concept Networks
Authors:
Aditya Gudimella,
Ross Story,
Matineh Shaker,
Ruofan Kong,
Matthew Brown,
Victor Shnayder,
Marcos Campos
Abstract:
Deep reinforcement learning yields great results for a large array of problems, but models are generally retrained anew for each new problem to be solved. Prior learning and knowledge are difficult to incorporate when training new models, requiring increasingly longer training as problems become more complex. This is especially problematic for problems with sparse rewards. We provide a solution to…
▽ More
Deep reinforcement learning yields great results for a large array of problems, but models are generally retrained anew for each new problem to be solved. Prior learning and knowledge are difficult to incorporate when training new models, requiring increasingly longer training as problems become more complex. This is especially problematic for problems with sparse rewards. We provide a solution to these problems by introducing Concept Network Reinforcement Learning (CNRL), a framework which allows us to decompose problems using a multi-level hierarchy. Concepts in a concept network are reusable, and flexible enough to encapsulate feature extractors, skills, or other concept networks. With this hierarchical learning approach, deep reinforcement learning can be used to solve complex tasks in a modular way, through problem decomposition. We demonstrate the strength of CNRL by training a model to grasp a rectangular prism and precisely stack it on top of a cube using a gripper on a Kinova JACO arm, simulated in MuJoCo. Our experiments show that our use of hierarchy results in a 45x reduction in environment interactions compared to the state-of-the-art on this task.
△ Less
Submitted 20 September, 2017;
originally announced September 2017.
