AI Hints
Statistics & Math for ML · Lesson 118 of 138

Practice: Exploring a Dataset

Describe, plot, question.

⏱ 2 min read▶ Video at 2:21:10🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Let's put the first half of the course to work. A cafe has six hundred receipts, with the day type, the temperature, and how much each customer spent. The owner asks, do people spend more at weekends? Let's explore first.

Practice: Exploring a Dataset
From the video: Practice: Exploring a Dataset
  • 600 cafe receipts
  • Day, temperature, spend
  • Do weekends earn more?

Describe first

Pandas describe gives the count, mean, standard deviation, and quartiles in one go. Then group by splits the receipts by day type. Run it. The mean spend is above the median, so spending is skewed right. And weekend customers spend about two and a half dollars more, on average.

explore.pypython
from cafe import df

print(df["spend"].describe().round(2).to_string())
print()
print(df.groupby("day")["spend"].agg(["count", "mean", "median"]).round(2))
Output
count    600.00
mean       8.23
std        3.60
min        2.44
25%        5.75
50%        7.56
75%        9.85
max       24.68

         count  mean  median
day
weekday    416  7.45    6.83
weekend    184  9.99    9.07
Also uses cafe.py
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 600
day = rng.choice(["weekday", "weekend"], n, p=[0.7, 0.3])
temp = rng.uniform(5, 30, n)
spend = rng.lognormal(1.8, 0.45, n) + np.where(day == "weekend", 2.0, 0) + 0.05 * temp
df = pd.DataFrame({"day": day, "temp": temp.round(1), "spend": spend.round(2)})

Then look

Numbers are a start, but always look at the shape. We draw a histogram of spend for each day type, on the same axes. Run it. Both are skewed right, with long tails, and the weekend histogram sits further to the right.

look.pypython
import matplotlib.pyplot as plt
from cafe import df

for day in ["weekday", "weekend"]:
    plt.hist(df[df["day"] == day]["spend"], bins=30, alpha=0.6, label=day)
plt.title("Spend per customer ($)"); plt.legend()
plt.savefig("spend.png")
Also uses cafe.py
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 600
day = rng.choice(["weekday", "weekend"], n, p=[0.7, 0.3])
temp = rng.uniform(5, 30, n)
spend = rng.lognormal(1.8, 0.45, n) + np.where(day == "weekend", 2.0, 0) + 0.05 * temp
df = pd.DataFrame({"day": day, "temp": temp.round(1), "spend": spend.round(2)})

Summary

Describe the data, group it, and look at its shape before testing anything. Next, testing the weekend question properly.

Key takeaways
  • describe()
  • groupby
  • Plot the shape
Next up: Practice: An Experiment End to End — from question to answer