Every example runs against the burglar alarms below: three clues per alarm, and the answer, whether it was real. A score depends on the data and the version, so the examples show what you would check rather than a long decimal.
The tables every example usesimport pandas as pd
from sklearn.tree import DecisionTreeClassifier
alarms = pd.DataFrame({
"sensors": [1, 4, 2, 5, 1, 3, 4, 1, 5, 2, 3, 1, 2, 4, 1, 5, 3, 2, 1, 4],
"door_forced": [0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0],
"hour": [14, 2, 11, 3, 16, 1, 23, 9, 2, 13, 22, 10, 4, 0, 15, 3, 12, 20, 9, 1],
"real": [0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1],
})
X = alarms[["sensors", "door_forced", "hour"]]
y = alarms["real"]X holds the clues, one row per alarm. y holds the answers, in the same order: 1 for a real break-in, 0 for a false alarm.
forkeywordslenbuilt-in functions"text"text42numbers# → 3what comes backClues and Answers
X and y
X is the clues, a table with one row per example. y is the answer for each row, a single column. Every scikit-learn model learns from this pair.
X.shape # → (20, 3)
len(y) # → 20The share of answers
For an answer of 0 and 1, the mean is the share of 1s: how rare the thing you are looking for is.
y.mean() # → 0.45A baseline first
What the laziest guess scores. A model is only worth what it beats this by.
always_no = (y == 0).mean()
always_no # → 0.55A First Model
Make, fit, predict
Make the model, .fit() it on the clues and answers, then .predict() an answer for each row of clues.
model = DecisionTreeClassifier(max_depth=3, random_state=0)
model.fit(X, y)
guesses = model.predict(X)model = DecisionTreeClassifier(max_depth=2, random_state=0)
model.fit(X, y)
guesses = model.predict(X)
len(guesses) # → 20Scoring by hand and .score()
The share of guesses that match the answers. .score() is the same sum in one call.
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
by_hand = (model.predict(X) == y).mean()
by_hand == model.score(X, y) # → TrueOne new row
Ask about tonight's alarm with a one-row table, the same columns in the same order.
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
tonight = pd.DataFrame({"sensors": [4], "door_forced": [1], "hour": [2]})
model.predict(tonight)[0] in (0, 1) # → Truemax_depth
How many questions in a row the tree may ask. No limit, and it can learn every row by heart and still guess new ones badly.
deep = DecisionTreeClassifier(random_state=0).fit(X, y)
deep.score(X, y) # → 1.0Testing Honestly
Visual reference
confusion_matrix(y_test, guesses): the real answers down the side, the guesses across the top.
Recall is caught out of every real one. Precision is caught out of every flag.
train_test_split()
Holds some rows back. The model learns on the rest and is marked on rows it never saw. The four pieces come back in this order.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
len(X_test) # → 5stratify=y
Keeps the share of each answer the same on both sides of the split, which matters when one answer is rare.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0, stratify=y)
abs(y_test.mean() - y.mean()) < 0.1 # → TrueDummyClassifier
The baseline as a model: strategy="most_frequent" always guesses the most common answer.
from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy="most_frequent").fit(X, y)
dummy.score(X, y) # → 0.55confusion_matrix()
A grid of right and wrong: the real answers down the side, the guesses across the top.
from sklearn.metrics import confusion_matrix
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
table = confusion_matrix(y, model.predict(X))
table.shape # → (2, 2)Reading a Tree
export_text()
The tree's rules written out as text, question by question, with the column names.
from sklearn.tree import export_text
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
print(export_text(model, feature_names=list(X.columns)))feature_importances_
How much the tree leaned on each clue, adding up to 1. Put it in a Series to see the names beside the numbers.
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
importance = pd.Series(model.feature_importances_, index=X.columns)
round(importance.sum(), 6) # → 1.0Predicting a Number
LinearRegression
A straight line through the data: .coef_ is how much the answer moves per unit of each clue, .intercept_ where it starts.
from sklearn.linear_model import LinearRegression
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 500]})
line = LinearRegression().fit(claims[["items"]], claims["claimed"])
round(line.coef_[0]) # → 96mean_absolute_error()
How far off the predictions are on average, in the answer's own units.
from sklearn.metrics import mean_absolute_error
mean_absolute_error([100, 200, 300], [110, 190, 330]) # → 16.666666666666668DummyRegressor
The number baseline: strategy="mean" guesses the average every time. Beat it or the model is not helping.
from sklearn.dummy import DummyRegressor
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 500]})
base = DummyRegressor(strategy="mean").fit(claims[["items"]], claims["claimed"])
base.predict(claims[["items"]])[0] # → 306.0A residual
What happened minus what the model expected. Sort by it, biggest first, to find what the model cannot explain.
from sklearn.linear_model import LinearRegression
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 900]})
line = LinearRegression().fit(claims[["items"]], claims["claimed"])
claims["gap"] = claims["claimed"] - line.predict(claims[["items"]])
claims.sort_values("gap", ascending=False)["items"].iloc[0] # → 5Rare Events
recall_score()
Of the real ones, the share the model caught. The number that matters when a miss is costly.
from sklearn.metrics import recall_score
recall_score([1, 1, 1, 1, 0, 0], [1, 1, 0, 0, 0, 1]) # → 0.5precision_score()
Of the ones it flagged, the share that were real. The number that matters when a false alarm is costly.
from sklearn.metrics import precision_score
round(precision_score([1, 1, 1, 1, 0, 0], [1, 1, 0, 0, 0, 1]), 2) # → 0.67predict_proba()
A chance for every row instead of a yes or no. [:, 1] keeps the chance of a 1.
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
chance = model.predict_proba(X)[:, 1]
len(chance) # → 20Moving the threshold
.predict() flags anything over a half. A lower line flags more: more caught, more false alarms.
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
chance = model.predict_proba(X)[:, 1]
(chance > 0.3).sum() >= (chance > 0.5).sum() # → TrueFair Scores
cross_val_score()
Five tests instead of one: the rows cut into five folds, each held back once. Report the average and look at the spread.
cross_val_score(model, X, y, cv=5, scoring="recall")from sklearn.model_selection import cross_val_score
scores = cross_val_score(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5)
len(scores) # → 5
spread = scores.max() - scores.min()scoring=
What each fold is marked on: "accuracy" by default, "recall", "precision", or "average_precision" for a model whose chances you will draw a line through.
from sklearn.model_selection import cross_val_score
ap = cross_val_score(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5, scoring="average_precision")
(ap >= 0).all() # → TrueHistGradientBoostingClassifier
Many small trees, each fixing the last one's mistakes. The model a data team would usually ship for tables like these.
from sklearn.ensemble import HistGradientBoostingClassifier
booster = HistGradientBoostingClassifier(random_state=0).fit(X, y)
len(booster.predict(X)) # → 20cross_val_predict()
A prediction for every row from a model that never saw that row: the fair way to give every row a chance.
from sklearn.model_selection import cross_val_predict
chance = cross_val_predict(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5, method="predict_proba")[:, 1]
len(chance) # → 20Words and Pipelines
pd.get_dummies()
A column of words into one True or False column per word, so a model can use it.
pays = pd.DataFrame({"amount": [20, 900, 45], "merchant": ["food", "giftcards", "food"]})
list(pd.get_dummies(pays, columns=["merchant"]).columns) # → ['amount', 'merchant_food', 'merchant_giftcards']Leakage
A clue that is only known after the answer, or IS the answer. A score that looks too good usually is.
leaky = alarms.assign(police_called=alarms["real"])
(leaky["police_called"] == leaky["real"]).mean() # → 1.0A time split
Train on the past and test on what came after, the way the model will really be used.
past = alarms.index < 15
X_past, y_past = X[past], y[past]
X_future, y_future = X[~past], y[~past]
len(X_future) # → 5OneHotEncoder and make_pipeline
An encoder that remembers the words it learned, and a pipeline that carries the preparation and the model as one object, so a raw row can go straight in.
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import make_pipeline
pays = pd.DataFrame({"amount": [20, 900, 45, 700], "merchant": ["food", "giftcards", "food", "giftcards"], "fraud": [0, 1, 0, 1]})
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["merchant"]), remainder="passthrough")
pipe = make_pipeline(prep, DecisionTreeClassifier(random_state=0))
pipe.fit(pays[["amount", "merchant"]], pays["fraud"])
pipe.predict(pd.DataFrame({"amount": [850], "merchant": ["giftcards"]}))[0] # → 1Explaining and Auditing
permutation_importance()
Shuffles one clue at a time and measures how far the score falls. A big fall means the model leans on that clue.
from sklearn.inspection import permutation_importance
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
result = permutation_importance(model, X, y, n_repeats=5, random_state=0)
len(result.importances_mean) # → 3False alarms by group
Among the honest rows only, the share flagged in each group. A gap means the model is harder on some people than others.
audit = pd.DataFrame({"district": ["A", "A", "B", "B"], "fraud": [0, 0, 0, 0], "flag": [True, False, False, False]})
honest = audit[audit["fraud"] == 0]
honest.groupby("district")["flag"].mean()["A"] # → 0.5IsolationForest
Finds odd rows with no answers at all. score_samples gives every row a score, lower for the odder ones.
from sklearn.ensemble import IsolationForest
claims = pd.DataFrame({"amount": [40, 42, 38, 41, 39, 400]})
finder = IsolationForest(random_state=0).fit(claims)
claims["score"] = finder.score_samples(claims)
claims.sort_values("score")["amount"].iloc[0] # → 400Which Score, When
| The question | The score | Call |
|---|---|---|
| How often is it right? | Accuracy | model.score(X_test, y_test) |
| Of the real ones, how many were caught? | Recall | recall_score(y_test, guesses) |
| Of the flags, how many were real? | Precision | precision_score(y_test, guesses) |
| How good are its chances, at every line? | Average precision | scoring="average_precision" |
| How far off is a number? | Mean absolute error | mean_absolute_error(y_test, predicted) |
Every score here is worth what it beats the baseline by. Score the dummy first.
