Title:A case for disaggregation of ML data processing

Abstract:Machine Learning (ML) computation requires feeding input data for the models to ingest. Traditionally, input data processing happens on the same host as the ML computation. The input data processing can however become a bottleneck of the ML computation if there are insufficient resources to process data quickly enough. This slows down the ML computation and wastes valuable and scarce ML hardware (e.g. GPUs and TPUs) used by the ML computation.
In this paper, we present this http URL service, a disaggregated input data processing service built on top of this http URL. Our work goes beyond describing the design and implementation of a new system which disaggregates preprocessing from ML computation and presents: (1) empirical evidence based on production workloads for the need of disaggregation, as well as quantitative evaluation of the impact disaggregation has on the performance and cost of production workloads, (2) benefits of disaggregation beyond horizontal scaling, (3) analysis of this http URL service's adoption at Google, the lessons learned during building and deploying the system and potential future lines of research opened up by our work.
We demonstrate that horizontally scaling data processing using this http URL service helps remove input bottlenecks, achieving speedups of up to 110x and job cost reductions of up to 89x. We further show that this http URL service can support computation reuse through data sharing across ML jobs with identical data processing pipelines (e.g. hyperparameter tuning jobs), incurring no performance penalty and reducing overall resource cost. Finally, we show that this http URL service advanced features can benefit performance of non-input bound jobs; in particular, coordinated data reads through this http URL service can yield up to 2x speedups and job cost savings for NLP jobs.