Automating EDA - Better Summaries from Pandas
The first thing that many of us do to quickly inspect a dataframe containing a fresh dataset is to use the
df.describe() method. This works well to quickly get an idea of the size and distribution of the
in the dataframe but often needs to be supplemented with additional code to check for outliers, correlations, or to
look at any
of the non-numerical data stored in the dataframe.
A while ago I discovered the package
pandas-profiling while browsing r/datascience and quickly realised
I could perform a lot of basic exploratory analysis in less then 3 lines of code. In this post I will describe the
key features of the package and
how to use it.
Jupyter and Virtual Environments
I start off every new project by creating a new virtual environment and by creating a new kernel for my notebooks.
$ virtualenv venv $ source venv/bin/activate $ pip install ipython $ ipython kernel install --user --name=pandas-profiling
For the purposes of this tutorial, I will be using the Titanic dataset to highlight the features available within pandas-profiling. This can be downloaded with
$ curl https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/stuff/titanic.csv -o titanic.csv
I then installed the necessary packages before launching a notebook
$ virtualenv venv $ source venv/bin/activate $ pip install ipython $ ipython kernel install --user --name=pandas-profiling $ jupyter notebook
After reading in your data into a dataframe, you can quickly build a report with
import pandas as pd import pandas_profiling df = pd.read_csv('./titanic.csv') profile = df.profile_report() profile
This outputs the report into a Jupyter cell. Since notebooks run in the browser, you can also easily export the report as a HTML file. Check out the full report - I will go through and describe the different sections below.
There are 5 main sections of the report. The first section is an overview of the dataset as a whole, giving you things like the number of samples, number of variables, variable types, and the size of data on the disk. There is also a subsection that highlights to the user some potential warnings such as columns with many zeroes, with a lot of missing data, duplicate records, or with many distinct values.
The second section goes through each of the columns, or variables, in the dataframe and provides summaries such as the mean, median, missing values, and a plot of the distribution.
A toggle allows the user to explore this variable in more depth giving you information on variance, measures of skew, and quartile information.
The third section gives information on the pairwise correlations of two columns. There are several measures including Person's r, Spearman's ρ, and for categorical columns there Cramér's V.
The final plot, and the penultimate section, is where you would the information on missing values and is presented in the form of a bar graph.
The last section gives two samples of the data, taken from the top and bottom of the dataset. These are equivalent
This tool has sped up a lot of my initial EDA work and presents the results in a nice and concise format to boot - would recommend!