MLPerf Training Benchmark
Authors:
Peter Mattson,
Christine Cheng,
Cody Coleman,
Greg Diamos,
Paulius Micikevicius,
David Patterson,
Hanlin Tang,
Gu-Yeon Wei,
Peter Bailis,
Victor Bittorf,
David Brooks,
Dehao Chen,
Debojyoti Dutta,
Udit Gupta,
Kim Hazelwood,
Andrew Hock,
Xinyuan Huang,
Atsushi Ike,
Bill Jia,
Daniel Kang,
David Kanter,
Naveen Kumar,
Jeffery Liao,
Guokai Ma,
Deepak Narayanan
, et al. (12 additional authors not shown)
Abstract:
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits h…
▽ More
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.
△ Less
Submitted 2 March, 2020; v1 submitted 2 October, 2019;
originally announced October 2019.
Scaling Up Collaborative Filtering Data Sets through Randomized Fractal Expansions
Authors:
Francois Belletti,
Karthik Lakshmanan,
Walid Krichene,
Nicolas Mayoraz,
Yi-Fan Chen,
John Anderson,
Taylor Robie,
Tayo Oguntebi,
Dan Shirron,
Amit Bleiwess
Abstract:
Recommender system research suffers from a disconnect between the size of academic data sets and the scale of industrial production systems. In order to bridge that gap, we propose to generate large-scale user/item interaction data sets by expanding pre-existing public data sets. Our key contribution is a technique that expands user/item incidence matrices matrices to large numbers of rows (users)…
▽ More
Recommender system research suffers from a disconnect between the size of academic data sets and the scale of industrial production systems. In order to bridge that gap, we propose to generate large-scale user/item interaction data sets by expanding pre-existing public data sets. Our key contribution is a technique that expands user/item incidence matrices matrices to large numbers of rows (users), columns (items), and non-zero values (interactions). The proposed method adapts Kronecker Graph Theory to preserve key higher order statistical properties such as the fat-tailed distribution of user engagements, item popularity, and singular value spectra of user/item interaction matrices. Preserving such properties is key to building large realistic synthetic data sets which in turn can be employed reliably to benchmark recommender systems and the systems employed to train them. We further apply our stochastic expansion algorithm to the binarized MovieLens 20M data set, which comprises 20M interactions between 27K movies and 138K users. The resulting expanded data set has 1.2B ratings, 2.2M users, and 855K items, which can be scaled up or down.
△ Less
Submitted 8 April, 2019;
originally announced May 2019.
