QueryCasequerycase.com
Lab reference guide

Python Statistics Cheat Sheet

The statistics the cases use, in pandas and scipy, with what each test is for and when to reach for it.

Every example runs against the two small tables below. A p-value or an interval depends on the data, so the examples show the comparison you would make (< 0.05) rather than a long decimal.

The tables every example uses
import pandas as pd
from scipy.stats import ttest_ind, chi2_contingency, pearsonr, linregress
trial = pd.DataFrame({
    "group": ["scheme"] * 6 + ["control"] * 6,
    "burglaries": [3, 5, 4, 2, 6, 4, 7, 6, 8, 5, 9, 7],
})
days = pd.DataFrame({
    "temp": [12, 15, 18, 21, 24, 27],
    "ice_cream": [40, 52, 71, 88, 104, 130],
    "assaults": [8, 10, 10, 13, 13, 16],
})

A patrol trial in trial, six wards given the scheme and six not, and six days of temperatures, ice cream sales and assaults in days.

forkeywordslenbuilt-in functions"text"text42numbers# → 3what comes back
01

The Middle and the Spread

mean and median

The mean adds everything up and shares it out; the median is the middle value. A few huge values drag the mean and leave the median alone.

Example
sentences = pd.Series([6, 8, 9, 12, 14, 180])
round(sentences.mean(), 1)  # → 38.2
sentences.median()  # → 10.5
TipWhen the mean and the median are far apart, the data has a long tail. Report the median as the typical value.

The share below a line

A comparison gives True or False per row, and the mean of True and False is the share that are True.

Example
sentences = pd.Series([6, 8, 9, 12, 14, 180])
round((sentences < sentences.mean()).mean(), 2)  # → 0.83

.std()

The standard deviation: roughly how far a typical value sits from the mean. Two groups can share a mean and differ wildly here.

Example
round(trial.groupby("group")["burglaries"].std()["scheme"], 2)  # → 1.41

.quantile()

The value a share of the data sits below. .quantile(0.9) is the 90th percentile, and two quantiles make a band.

Example
readings = pd.Series([31, 33, 34, 35, 35, 36, 38, 41])
readings.quantile(0.5)  # → 35.0
round(readings.quantile(0.9) - readings.quantile(0.1), 1)  # → 6.5

A z-score

How many standard deviations a value sits from the mean. Far past 3 either way is worth a second look.

Pattern
(column - column.mean()) / column.std()
Example
speeds = pd.Series([31, 29, 30, 33, 28, 30, 32, 29, 31, 412])
z = (speeds - speeds.mean()) / speeds.std()
(z > 2).sum()  # → 1
02

Samples and Intervals

.sample()

Draws rows at random. random_state fixes the draw so it can be repeated exactly.

Pattern
column.sample(n, random_state=0)
Example
pick = trial["burglaries"].sample(4, random_state=0)
len(pick)  # → 4

Many samples

Draw again and again in a loop to see how far one sample's answer can land from the truth.

Example
means = []
for i in range(1000):
    means.append(trial["burglaries"].sample(6, random_state=i).mean())
len(means)  # → 1000

The bootstrap

When there is only one sample, resample IT with replace=True: the same size, rows allowed to repeat.

Example
scheme = trial[trial["group"] == "scheme"]["burglaries"]
boots = []
for i in range(1000):
    boots.append(scheme.sample(6, replace=True, random_state=i).mean())
low = pd.Series(boots).quantile(0.025)
high = pd.Series(boots).quantile(0.975)
low < scheme.mean() < high  # → True

A 95% interval

The middle 95% of the resampled means, from the 2.5th to the 97.5th percentile: a range the true value probably sits in.

Pattern
pd.Series(values).quantile(0.025), pd.Series(values).quantile(0.975)

The standard error

How far a sample's mean typically lands from the truth: the spread divided by the square root of the sample size.

Pattern
sample.std() / len(sample) ** 0.5
Example
scheme = trial[trial["group"] == "scheme"]["burglaries"]
se = scheme.std() / len(scheme) ** 0.5
round(se, 2)  # → 0.58
TipTen times the sample size buys about three times the precision, not ten: the square root.
03

Comparing Two Groups

The gap between two groups

A groupby gives each group's mean, labelled by group, and a label in square brackets picks one out.

Example
means = trial.groupby("group")["burglaries"].mean()
means["control"] - means["scheme"]  # → 3.0

Shuffled labels

Shuffle the group labels and measure the gap again: what chance alone does. ignore_index=True stops each label sliding back to its own row.

Example
trial["fake"] = trial["group"].sample(frac=1, random_state=0, ignore_index=True)
trial.groupby("fake")["burglaries"].mean()

A p-value from shuffles

The share of shuffled gaps at least as big as the real one. Small means chance rarely does as well.

Example
gaps = []
for i in range(1000):
    fake = trial["group"].sample(frac=1, random_state=i, ignore_index=True)
    m = trial.groupby(fake)["burglaries"].mean()
    gaps.append(m["control"] - m["scheme"])
p = (pd.Series(gaps).abs() >= 3.0).mean()
p < 0.05  # → True
Tip.abs() counts a gap either way, which is the two-sided test. Use it unless you said which way before you looked.

ttest_ind()

The textbook test of two groups' means in one line. equal_var=False (Welch's test) does not assume the two spread the same way.

Pattern
ttest_ind(group_a, group_b, equal_var=False)
Example
scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
result = ttest_ind(control, scheme, equal_var=False)
result.pvalue < 0.05  # → True

.confidence_interval()

The range the true gap probably sits in, 95% by default. An interval that clears 0 says the gap is real; how far it clears it says how big.

Example
scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
ci = ttest_ind(control, scheme, equal_var=False).confidence_interval()
ci.low > 0  # → True
04

Counts and Proportions

A rate per group

For a column of 0 and 1, the mean is the rate: the share of rows that are 1.

Example
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
letters.groupby("group")["again"].mean()["letter"]  # → 0.2

pd.crosstab()

A grid of counts: one row per value of the first column, one column per value of the second.

Example
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
table = pd.crosstab(letters["group"], letters["again"])
table.loc["none", 1]  # → 4

normalize="index"

Turns each row of a crosstab into shares that add up to 1, so rows of different sizes can be compared.

Example
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
pd.crosstab(letters["group"], letters["again"], normalize="index").loc["none", 1]  # → 0.8

chi2_contingency()

Tests a grid of counts against what no link at all would give. .expected_freq is that no-link grid.

Example
table = pd.DataFrame({"no": [80, 60], "yes": [20, 40]}, index=["letter", "none"])
result = chi2_contingency(table)
result.pvalue < 0.05  # → True
result.expected_freq[0][0]  # → 70.0
05

Many Tests and Power

Many tests at once

Run the same test on every group in a loop and keep each p-value. With 20 tests, about one clears 0.05 by luck alone.

Example
pvalues = {"Ashby": 0.03, "Dunmore": 0.4, "Kelby": 0.2, "Pellow": 0.001}
(pd.Series(pvalues) < 0.05).sum()  # → 2

A bar raised for many tests

Divide 0.05 by the number of tests (Bonferroni), so the chance of any false alarm across all of them stays near 5%.

Example
pvalues = {"Ashby": 0.03, "Dunmore": 0.4, "Kelby": 0.2, "Pellow": 0.001}
(pd.Series(pvalues) < 0.05 / len(pvalues)).sum()  # → 1

Power, by simulation

How often a trial of a given size finds an effect that is really there: run the small trial many times and count the wins.

Example
scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
hits = []
for i in range(200):
    s = scheme.sample(4, replace=True, random_state=i)
    c = control.sample(4, replace=True, random_state=i + 1000)
    hits.append(ttest_ind(c, s, equal_var=False).pvalue < 0.05)
power = pd.Series(hits).mean()
TipA trial too small to find the effect will usually say "no significant effect". That is a statement about the trial, not the scheme.
07

Fair Comparisons

Rates within each kind

Group by two columns to compare like with like. A headline rate can reverse once each kind is compared on its own.

Example
cases = pd.DataFrame({"offence": ["theft"] * 4 + ["assault"] * 4, "process": ["new", "old"] * 4, "convicted": [1, 1, 1, 0, 0, 0, 1, 0]})
cases.groupby(["offence", "process"])["convicted"].mean()

A rate, not a count

Divide by how much there was to count: incidents per visitor, not incidents. A busy place has more of everything.

Example
stations = pd.DataFrame({"incidents": [16, 6], "footfall": [40, 10]})
stations["rate"] = stations["incidents"] / stations["footfall"]
stations["rate"].tolist()  # → [0.4, 0.6]

What would have happened anyway

Fit the line on the places left alone, then ask it about the places that were changed. The gap between the line and what happened is the effect.

Example
others = pd.DataFrame({"year1": [2, 4, 5, 7], "year2": [3, 4, 5, 6]})
fit = linregress(others["year1"], others["year2"])
expected = fit.intercept + fit.slope * 10
round(expected, 1)  # → 7.9
TipThe worst year on record is usually followed by a better one, change or no change. That is regression to the mean.
08

Which Test, When

The questionThe callRead
Do two groups' averages differ?ttest_ind(a, b, equal_var=False).pvalue, .confidence_interval()
Is a grid of counts linked?chi2_contingency(table).pvalue, .expected_freq
Do two columns move together?pearsonr(x, y).statistic, .pvalue
How much does y change with x?linregress(x, y).slope, .intercept, .rvalue ** 2
Could chance alone do this?Shuffle the labels 1,000 timesThe share of shuffled gaps as big
How sure is one sample?Bootstrap 1,000 resamples.quantile(0.025) and .quantile(0.975)

A p-value under 0.05 is "statistically significant" by convention. It says the result would be surprising if nothing were going on, not how big or how important it is.

Python Statistics Cheat Sheet · QueryCaseThe full course is at querycase.com