Every example runs against the two small tables below. A p-value or an interval depends on the data, so the examples show the comparison you would make (< 0.05) rather than a long decimal.
import pandas as pd
from scipy.stats import ttest_ind, chi2_contingency, pearsonr, linregress
trial = pd.DataFrame({
"group": ["scheme"] * 6 + ["control"] * 6,
"burglaries": [3, 5, 4, 2, 6, 4, 7, 6, 8, 5, 9, 7],
})
days = pd.DataFrame({
"temp": [12, 15, 18, 21, 24, 27],
"ice_cream": [40, 52, 71, 88, 104, 130],
"assaults": [8, 10, 10, 13, 13, 16],
})A patrol trial in trial, six wards given the scheme and six not, and six days of temperatures, ice cream sales and assaults in days.
forkeywordslenbuilt-in functions"text"text42numbers# → 3what comes backThe Middle and the Spread
mean and median
The mean adds everything up and shares it out; the median is the middle value. A few huge values drag the mean and leave the median alone.
sentences = pd.Series([6, 8, 9, 12, 14, 180])
round(sentences.mean(), 1) # → 38.2
sentences.median() # → 10.5The share below a line
A comparison gives True or False per row, and the mean of True and False is the share that are True.
sentences = pd.Series([6, 8, 9, 12, 14, 180])
round((sentences < sentences.mean()).mean(), 2) # → 0.83.std()
The standard deviation: roughly how far a typical value sits from the mean. Two groups can share a mean and differ wildly here.
round(trial.groupby("group")["burglaries"].std()["scheme"], 2) # → 1.41.quantile()
The value a share of the data sits below. .quantile(0.9) is the 90th percentile, and two quantiles make a band.
readings = pd.Series([31, 33, 34, 35, 35, 36, 38, 41])
readings.quantile(0.5) # → 35.0
round(readings.quantile(0.9) - readings.quantile(0.1), 1) # → 6.5A z-score
How many standard deviations a value sits from the mean. Far past 3 either way is worth a second look.
(column - column.mean()) / column.std()speeds = pd.Series([31, 29, 30, 33, 28, 30, 32, 29, 31, 412])
z = (speeds - speeds.mean()) / speeds.std()
(z > 2).sum() # → 1Samples and Intervals
.sample()
Draws rows at random. random_state fixes the draw so it can be repeated exactly.
column.sample(n, random_state=0)pick = trial["burglaries"].sample(4, random_state=0)
len(pick) # → 4Many samples
Draw again and again in a loop to see how far one sample's answer can land from the truth.
means = []
for i in range(1000):
means.append(trial["burglaries"].sample(6, random_state=i).mean())
len(means) # → 1000The bootstrap
When there is only one sample, resample IT with replace=True: the same size, rows allowed to repeat.
scheme = trial[trial["group"] == "scheme"]["burglaries"]
boots = []
for i in range(1000):
boots.append(scheme.sample(6, replace=True, random_state=i).mean())
low = pd.Series(boots).quantile(0.025)
high = pd.Series(boots).quantile(0.975)
low < scheme.mean() < high # → TrueA 95% interval
The middle 95% of the resampled means, from the 2.5th to the 97.5th percentile: a range the true value probably sits in.
pd.Series(values).quantile(0.025), pd.Series(values).quantile(0.975)The standard error
How far a sample's mean typically lands from the truth: the spread divided by the square root of the sample size.
sample.std() / len(sample) ** 0.5scheme = trial[trial["group"] == "scheme"]["burglaries"]
se = scheme.std() / len(scheme) ** 0.5
round(se, 2) # → 0.58Comparing Two Groups
The gap between two groups
A groupby gives each group's mean, labelled by group, and a label in square brackets picks one out.
means = trial.groupby("group")["burglaries"].mean()
means["control"] - means["scheme"] # → 3.0Shuffled labels
Shuffle the group labels and measure the gap again: what chance alone does. ignore_index=True stops each label sliding back to its own row.
trial["fake"] = trial["group"].sample(frac=1, random_state=0, ignore_index=True)
trial.groupby("fake")["burglaries"].mean()A p-value from shuffles
The share of shuffled gaps at least as big as the real one. Small means chance rarely does as well.
gaps = []
for i in range(1000):
fake = trial["group"].sample(frac=1, random_state=i, ignore_index=True)
m = trial.groupby(fake)["burglaries"].mean()
gaps.append(m["control"] - m["scheme"])
p = (pd.Series(gaps).abs() >= 3.0).mean()
p < 0.05 # → True.abs() counts a gap either way, which is the two-sided test. Use it unless you said which way before you looked.ttest_ind()
The textbook test of two groups' means in one line. equal_var=False (Welch's test) does not assume the two spread the same way.
ttest_ind(group_a, group_b, equal_var=False)scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
result = ttest_ind(control, scheme, equal_var=False)
result.pvalue < 0.05 # → True.confidence_interval()
The range the true gap probably sits in, 95% by default. An interval that clears 0 says the gap is real; how far it clears it says how big.
scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
ci = ttest_ind(control, scheme, equal_var=False).confidence_interval()
ci.low > 0 # → TrueCounts and Proportions
A rate per group
For a column of 0 and 1, the mean is the rate: the share of rows that are 1.
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
letters.groupby("group")["again"].mean()["letter"] # → 0.2pd.crosstab()
A grid of counts: one row per value of the first column, one column per value of the second.
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
table = pd.crosstab(letters["group"], letters["again"])
table.loc["none", 1] # → 4normalize="index"
Turns each row of a crosstab into shares that add up to 1, so rows of different sizes can be compared.
letters = pd.DataFrame({"group": ["letter"] * 5 + ["none"] * 5, "again": [0, 0, 1, 0, 0, 1, 1, 0, 1, 1]})
pd.crosstab(letters["group"], letters["again"], normalize="index").loc["none", 1] # → 0.8chi2_contingency()
Tests a grid of counts against what no link at all would give. .expected_freq is that no-link grid.
table = pd.DataFrame({"no": [80, 60], "yes": [20, 40]}, index=["letter", "none"])
result = chi2_contingency(table)
result.pvalue < 0.05 # → True
result.expected_freq[0][0] # → 70.0Many Tests and Power
Many tests at once
Run the same test on every group in a loop and keep each p-value. With 20 tests, about one clears 0.05 by luck alone.
pvalues = {"Ashby": 0.03, "Dunmore": 0.4, "Kelby": 0.2, "Pellow": 0.001}
(pd.Series(pvalues) < 0.05).sum() # → 2A bar raised for many tests
Divide 0.05 by the number of tests (Bonferroni), so the chance of any false alarm across all of them stays near 5%.
pvalues = {"Ashby": 0.03, "Dunmore": 0.4, "Kelby": 0.2, "Pellow": 0.001}
(pd.Series(pvalues) < 0.05 / len(pvalues)).sum() # → 1Power, by simulation
How often a trial of a given size finds an effect that is really there: run the small trial many times and count the wins.
scheme = trial[trial["group"] == "scheme"]["burglaries"]
control = trial[trial["group"] == "control"]["burglaries"]
hits = []
for i in range(200):
s = scheme.sample(4, replace=True, random_state=i)
c = control.sample(4, replace=True, random_state=i + 1000)
hits.append(ttest_ind(c, s, equal_var=False).pvalue < 0.05)
power = pd.Series(hits).mean()Links Between Columns
.corr()
How closely two columns move together, from -1 to 1. On a whole table it gives every pair at once.
round(days["ice_cream"].corr(days["assaults"]), 2) # → 0.97
days[["temp", "ice_cream", "assaults"]].corr()pearsonr()
The same correlation, with a p-value: whether a link this strong could be chance.
result = pearsonr(days["ice_cream"], days["assaults"])
result.pvalue < 0.05 # → TrueHolding a third column still
Keep only rows where the third column barely moves, and see what is left of the link.
mild = days[(days["temp"] >= 15) & (days["temp"] < 25)]
len(mild) # → 4linregress()
The straight line through two columns: its slope, where it crosses, and rvalue ** 2, the share of the movement it explains.
fit = linregress(days["temp"], days["assaults"])
round(fit.slope, 2) # → 0.5
round(fit.intercept + fit.slope * 25, 1) # → 14.4A line on two columns
scikit-learn's LinearRegression with two clue columns gives each one's effect with the other held still.
from sklearn.linear_model import LinearRegression
both = LinearRegression().fit(days[["ice_cream", "temp"]], days["assaults"])
len(both.coef_) # → 2Fair Comparisons
Rates within each kind
Group by two columns to compare like with like. A headline rate can reverse once each kind is compared on its own.
cases = pd.DataFrame({"offence": ["theft"] * 4 + ["assault"] * 4, "process": ["new", "old"] * 4, "convicted": [1, 1, 1, 0, 0, 0, 1, 0]})
cases.groupby(["offence", "process"])["convicted"].mean()A rate, not a count
Divide by how much there was to count: incidents per visitor, not incidents. A busy place has more of everything.
stations = pd.DataFrame({"incidents": [16, 6], "footfall": [40, 10]})
stations["rate"] = stations["incidents"] / stations["footfall"]
stations["rate"].tolist() # → [0.4, 0.6]What would have happened anyway
Fit the line on the places left alone, then ask it about the places that were changed. The gap between the line and what happened is the effect.
others = pd.DataFrame({"year1": [2, 4, 5, 7], "year2": [3, 4, 5, 6]})
fit = linregress(others["year1"], others["year2"])
expected = fit.intercept + fit.slope * 10
round(expected, 1) # → 7.9Which Test, When
| The question | The call | Read |
|---|---|---|
| Do two groups' averages differ? | ttest_ind(a, b, equal_var=False) | .pvalue, .confidence_interval() |
| Is a grid of counts linked? | chi2_contingency(table) | .pvalue, .expected_freq |
| Do two columns move together? | pearsonr(x, y) | .statistic, .pvalue |
| How much does y change with x? | linregress(x, y) | .slope, .intercept, .rvalue ** 2 |
| Could chance alone do this? | Shuffle the labels 1,000 times | The share of shuffled gaps as big |
| How sure is one sample? | Bootstrap 1,000 resamples | .quantile(0.025) and .quantile(0.975) |
A p-value under 0.05 is "statistically significant" by convention. It says the result would be surprising if nothing were going on, not how big or how important it is.
