Matrix Factorization at Scale: a Comparison of Scientific Data Analytics in Spark and C+MPI Using Three Case Studies
Authors:
Alex Gittens,
Aditya Devarakonda,
Evan Racah,
Michael Ringenburg,
Lisa Gerhardt,
Jey Kottalam,
Jialin Liu,
Kristyn Maschhoff,
Shane Canon,
Jatin Chhugani,
Pramod Sharma,
Jiyan Yang,
James Demmel,
Jim Harrell,
Venkat Krishnamurthy,
Michael W. Mahoney,
Prabhat
Abstract:
We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks. We examine three widely-used and important matrix factorizations: NMF (for physical plausability), PCA (for its ubiquity…
▽ More
We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks. We examine three widely-used and important matrix factorizations: NMF (for physical plausability), PCA (for its ubiquity) and CX (for data interpretability). We apply these methods to TB-sized problems in particle physics, climate modeling and bioimaging. The data matrices are tall-and-skinny which enable the algorithms to map conveniently into Spark's data-parallel model. We perform scaling experiments on up to 1600 Cray XC40 nodes, describe the sources of slowdowns, and provide tuning guidance to obtain high performance.
△ Less
Submitted 20 September, 2016; v1 submitted 5 July, 2016;
originally announced July 2016.
Fast Updates on Read-Optimized Databases Using Multi-Core CPUs
Authors:
Jens Krueger,
Changkyu Kim,
Martin Grund,
Nadathur Satish,
David Schwalb,
Jatin Chhugani,
Hasso Plattner,
Pradeep Dubey,
Alexander Zeier
Abstract:
Read-optimized columnar databases use differential updates to handle writes by maintaining a separate write-optimized delta partition which is periodically merged with the read-optimized and compressed main partition. This merge process introduces significant overheads and unacceptable downtimes in update intensive systems, aspiring to combine transactional and analytical workloads into one system…
▽ More
Read-optimized columnar databases use differential updates to handle writes by maintaining a separate write-optimized delta partition which is periodically merged with the read-optimized and compressed main partition. This merge process introduces significant overheads and unacceptable downtimes in update intensive systems, aspiring to combine transactional and analytical workloads into one system. In the first part of the paper, we report data analyses of 12 SAP Business Suite customer systems. In the second half, we present an optimized merge process reducing the merge overhead of current systems by a factor of 30. Our linear-time merge algorithm exploits the underlying high compute and bandwidth resources of modern multi-core CPUs with architecture-aware optimizations and efficient parallelization. This enables compressed in-memory column stores to handle the transactional update rate required by enterprise applications, while keeping properties of read-optimized databases for analytic-style queries.
△ Less
Submitted 30 September, 2011;
originally announced September 2011.
