Showing 1–1 of 1 results for author: Tarn, E

Serverless inferencing on Kubernetes

Authors: Clive Cox, Dan Sun, Ellis Tarn, Animesh Singh, Rakesh Kelkar, David Goodwin

Abstract: Organisations are increasingly putting machine learning models into production at scale. The increasing popularity of serverless scale-to-zero paradigms presents an opportunity for deploying machine learning models to help mitigate infrastructure costs when many models may not be in continuous use. We will discuss the KFServing project which builds on the KNative serverless paradigm to provide a s… ▽ More Organisations are increasingly putting machine learning models into production at scale. The increasing popularity of serverless scale-to-zero paradigms presents an opportunity for deploying machine learning models to help mitigate infrastructure costs when many models may not be in continuous use. We will discuss the KFServing project which builds on the KNative serverless paradigm to provide a serverless machine learning inference solution that allows a consistent and simple interface for data scientists to deploy their models. We will show how it solves the challenges of autoscaling GPU based inference and discuss some of the lessons learnt from using it in production. △ Less

Submitted 24 July, 2020; v1 submitted 14 July, 2020; originally announced July 2020.

Comments: 4 pages, 1 figure, presented at workshop on "Challenges in Deploying and Monitoring Machine Learning System" at ICML 2020