Title:DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python

Abstract:Exploratory Data Analysis (EDA) is a crucial step in any data science project. However, existing Python libraries fall short in supporting data scientists to complete common EDA tasks for statistical modeling. Their API design is either too low level, which is optimized for plotting rather than EDA, or too high level, which is hard to specify more fine-grained EDA tasks. In response, we propose this http URL, a novel task-centric EDA system in Python. this http URL allows data scientists to declaratively specify a wide range of EDA tasks in different granularity with a single function call. We identify a number of challenges to implement this http URL, and propose effective solutions to improve the scalability, usability, customizability of the system. In particular, we discuss some lessons learned from using Dask to build the data processing pipelines for EDA tasks and describe our approaches to accelerate the pipelines. We conduct extensive experiments to compare this http URL with Pandas-profiling, the state-of-the-art EDA system in Python. The experiments show that this http URL significantly outperforms Pandas-profiling in terms of both speed and user experience. this http URL is open-sourced as an EDA component of DataPrep: this https URL.