Showing 1–4 of 4 results for author: Colmenares, J A

Admission Control with Response Time Objectives for Low-latency Online Data Systems

Authors: Hao Xu, Juan A. Colmenares

Abstract: To provide quick responses to users, Internet companies rely on online data systems able to answer queries in milliseconds. These systems employ complementary overload management techniques to ensure they provide a continued, acceptable service through-out traffic surges, where 'acceptable' partly means that serviced queries meet or track closely their response time objectives. Thus, in this paper… ▽ More To provide quick responses to users, Internet companies rely on online data systems able to answer queries in milliseconds. These systems employ complementary overload management techniques to ensure they provide a continued, acceptable service through-out traffic surges, where 'acceptable' partly means that serviced queries meet or track closely their response time objectives. Thus, in this paper we present Bouncer, an admission control policy aimed to keep admitted queries under or near their service level objectives (SLOs) on percentile response times. It computes inexpensive estimates of percentile response times for every incoming query and compares the estimates against the objective values to decide whether to accept or reject the query. Bouncer allows assigning separate SLOs to different classes of queries in the workload, implements early rejections to let clients react promptly and to help data systems avoid doing useless work, and complements other load shedding policies that guard systems from exceeding their capacity. Moreover, we propose two starvation avoidance strategies that supplement Bouncer's basic formulation and prevent query types from receiving no service (starving). We evaluate Bouncer and its starvation-avoiding variants against other policies in simulation and on a production-grade in-memory distributed graph database. Our results show that Bouncer and its variants allow admitted queries to meet or stay close to the SLOs when the other policies do not. They also report fewer overall rejections, a small overhead, and with the given latency SLOs, they let the system reach high utilization. In addition, we observe that the proposed strategies can stop query starvation, but at the expense of a modest increase in overall rejections and causing SLO violations for serviced requests. △ Less

Submitted 22 December, 2023; originally announced December 2023.

What's the Problem, Linda? The Conjunction Fallacy as a Fairness Problem

Authors: Jose Alvarez Colmenares

Abstract: The field of Artificial Intelligence (AI) is focusing on creating automated decision-making (ADM) systems that operate as close as possible to human-like intelligence. This effort has pushed AI researchers into exploring cognitive fields like psychology. The work of Daniel Kahneman and the late Amos Tversky on biased human decision-making, including the study of the conjunction fallacy, has experi… ▽ More The field of Artificial Intelligence (AI) is focusing on creating automated decision-making (ADM) systems that operate as close as possible to human-like intelligence. This effort has pushed AI researchers into exploring cognitive fields like psychology. The work of Daniel Kahneman and the late Amos Tversky on biased human decision-making, including the study of the conjunction fallacy, has experienced a second revival because of this. Under the conjunction fallacy a human decision-maker will go against basic probability laws and rank as more likely a conjunction over one of its parts. It has been proven overtime through a set of experiments with the Linda Problem being the most famous one. Although this interdisciplinary effort is welcomed, we fear that AI researchers ignore the driving force behind the conjunction fallacy as captured by the Linda Problem: the fact that Linda must be stereotypically described as a woman. In this paper we revisit the Linda Problem and formulate it as a fairness problem. In doing so we introduce perception as a parameter of interest through the structural causal perception framework. Using an illustrative decision-making example, we showcase the proposed conceptual framework and its potential impact for developing fair ADM systems. △ Less

Submitted 28 March, 2024; v1 submitted 16 May, 2023; originally announced May 2023.

Comments: Large portions of this paper were used for another paper with a different direction. Therefore, this paper became an early draft of the other paper, which is why I am removing it until it can stand by itself

Ingestion, Indexing and Retrieval of High-Velocity Multidimensional Sensor Data on a Single Node

Authors: Juan A. Colmenares, Reza Dorrigiv, Daniel G. Waddington

Abstract: Multidimensional data are becoming more prevalent, partly due to the rise of the Internet of Things (IoT), and with that the need to ingest and analyze data streams at rates higher than before. Some industrial IoT applications require ingesting millions of records per second, while processing queries on recently ingested and historical data. Unfortunately, existing database systems suited to multi… ▽ More Multidimensional data are becoming more prevalent, partly due to the rise of the Internet of Things (IoT), and with that the need to ingest and analyze data streams at rates higher than before. Some industrial IoT applications require ingesting millions of records per second, while processing queries on recently ingested and historical data. Unfortunately, existing database systems suited to multidimensional data exhibit low per-node ingestion performance, and even if they can scale horizontally in distributed settings, they require large number of nodes to meet such ingest demands. For this reason, in this paper we evaluate a single-node multidimensional data store for high-velocity sensor data. Its design centers around a two-level indexing structure, wherein the global index is an in-memory R*-tree and the local indices are serialized kd-trees. This study is confined to records with numerical indexing fields and range queries, and covers ingest throughput, query response time, and storage footprint. We show that the adopted design streamlines data ingestion and offers ingress rates two orders of magnitude higher than those of Percona Server, SQLite, and Druid. Our prototype also reports query response times comparable to or better than those of Percona Server and Druid, and compares favorably in terms of storage footprint. In addition, we evaluate a kd-tree partitioning based scheme for grouping incoming streamed data records. Compared to a random scheme, this scheme produces less overlap between groups of streamed records, but contrary to what we expected, such reduced overlap does not translate into better query performance. By contrast, the local indices prove much more beneficial to query performance. We believe the experience reported in this paper is valuable to practitioners and researchers alike interested in building database systems for high-velocity multidimensional data. △ Less

Submitted 4 July, 2017; originally announced July 2017.

Efficient Detection of Points of Interest from Georeferenced Visual Content

Authors: Ying Lu, Juan A. Colmenares

Abstract: Many people take photos and videos with smartphones and more recently with 360-degree cameras at popular places and events, and share them in social media. Such visual content is produced in large volumes in urban areas, and it is a source of information that online users could exploit to learn what has got the interest of the general public on the streets of the cities where they live or plan to… ▽ More Many people take photos and videos with smartphones and more recently with 360-degree cameras at popular places and events, and share them in social media. Such visual content is produced in large volumes in urban areas, and it is a source of information that online users could exploit to learn what has got the interest of the general public on the streets of the cities where they live or plan to visit. A key step to providing users with that information is to identify the most popular k spots in specified areas. In this paper, we propose a clustering and incremental sampling (C&IS) approach that trades off accuracy of top-k results for detection speed. It uses clustering to determine areas with high density of visual content, and incremental sampling, controlled by stopping criteria, to limit the amount of computational work. It leverages spatial metadata, which represent the scenes in the visual content, to rapidly detect the hotspots, and uses a recently proposed Gaussian probability model to describe the capture intention distribution in the query area. We evaluate the approach with metadata, derived from a non-synthetic, user-generated dataset, for regular mobile and 360-degree visual content. Our results show that the C&IS approach offers 2.8x-19x reductions in processing time over an optimized baseline, while in most cases correctly identifying 4 out of 5 top locations. △ Less

Submitted 2 June, 2017; originally announced June 2017.