Extending Polaris to Support Transactions
Authors:
Josep Aguilar-Saborit,
Raghu Ramakrishnan,
Kevin Bocksrocker,
Alan Halverson,
Konstantin Kosinsky,
Ryan O'Connor,
Nadejda Poliakova,
Moe Shafiei,
Taewoo Kim,
Phil Kon-Kim,
Haris Mahmud-Ansari,
Blazej Matuszyk,
Matt Miles,
Sumin Mohanan,
Cristian Petculescu,
Ishan Rahesh-Madan,
Emma Rose-Wirshing,
Elias Yousefi
Abstract:
In Polaris, we introduced a cloud-native distributed query processor to perform analytics at scale. In this paper, we extend the underlying Polaris distributed computation framework, which can be thought of as a read-only transaction engine, to execute general transactions (including updates, deletes, inserts and bulk loads, in addition to queries) for Tier 1 warehousing workloads in a highly perf…
▽ More
In Polaris, we introduced a cloud-native distributed query processor to perform analytics at scale. In this paper, we extend the underlying Polaris distributed computation framework, which can be thought of as a read-only transaction engine, to execute general transactions (including updates, deletes, inserts and bulk loads, in addition to queries) for Tier 1 warehousing workloads in a highly performant and predictable manner. We take advantage of the immutability of data files in log-structured data stores and build on SQL Server transaction management to deliver full transactional support with Snapshot Isolation semantics, including multi-table and multi-statement transactions. With the enhancements described in this paper, Polaris supports both query processing and transactions for T-SQL in Microsoft Fabric.
△ Less
Submitted 20 January, 2024;
originally announced January 2024.
LST-Bench: Benchmarking Log-Structured Tables in the Cloud
Authors:
Jesús Camacho-Rodríguez,
Ashvin Agrawal,
Anja Gruenheid,
Ashit Gosalia,
Cristian Petculescu,
Josep Aguilar-Saborit,
Avrilia Floratou,
Carlo Curino,
Raghu Ramakrishnan
Abstract:
Data processing engines increasingly leverage distributed file systems for scalable, cost-effective storage. While the Apache Parquet columnar format has become a popular choice for data storage and retrieval, the immutability of Parquet files renders it impractical to meet the demands of frequent updates in contemporary analytical workloads. Log-Structured Tables (LSTs), such as Delta Lake, Apach…
▽ More
Data processing engines increasingly leverage distributed file systems for scalable, cost-effective storage. While the Apache Parquet columnar format has become a popular choice for data storage and retrieval, the immutability of Parquet files renders it impractical to meet the demands of frequent updates in contemporary analytical workloads. Log-Structured Tables (LSTs), such as Delta Lake, Apache Iceberg, and Apache Hudi, offer an alternative for scenarios requiring data mutability, providing a balance between efficient updates and the benefits of columnar storage. They provide features like transactions, time-travel, and schema evolution, enhancing usability and enabling access from multiple engines. Moreover, engines like Apache Spark and Trino can be configured to leverage the optimizations and controls offered by LSTs to meet specific business needs. Conventional benchmarks and tools are inadequate for evaluating the transformative changes in the storage layer resulting from these advancements, as they do not allow us to measure the impact of design and optimization choices in this new setting.
In this paper, we propose a novel benchmarking approach and metrics that build upon existing benchmarks, aiming to systematically assess LSTs. We develop a framework, LST-Bench, which facilitates effective exploration and evaluation of the collaborative functioning of LSTs and data processing engines through tailored benchmark packages. A package is a mix of use patterns reflecting a target workload; LST-Bench makes it easy to define a wide range of use patterns and combine them into a package, and we include a baseline package for completeness. Our assessment demonstrates the effectiveness of our framework and benchmark packages in extracting valuable insights across diverse environments. The code for LST-Bench is open-sourced and is available at https://github.com/microsoft/lst-bench/ .
△ Less
Submitted 19 January, 2024; v1 submitted 1 May, 2023;
originally announced May 2023.
