QueryCasequerycase.com
Lab reference guide

Scikit-learn Cheat Sheet

Machine learning the way the cases teach it: a baseline first, an honest test, and a score that means what it says.

Every example runs against the burglar alarms below: three clues per alarm, and the answer, whether it was real. A score depends on the data and the version, so the examples show what you would check rather than a long decimal.

The tables every example uses
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
alarms = pd.DataFrame({
    "sensors":     [1, 4, 2, 5, 1, 3, 4, 1, 5, 2, 3, 1, 2, 4, 1, 5, 3, 2, 1, 4],
    "door_forced": [0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0],
    "hour":        [14, 2, 11, 3, 16, 1, 23, 9, 2, 13, 22, 10, 4, 0, 15, 3, 12, 20, 9, 1],
    "real":        [0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1],
})
X = alarms[["sensors", "door_forced", "hour"]]
y = alarms["real"]

X holds the clues, one row per alarm. y holds the answers, in the same order: 1 for a real break-in, 0 for a false alarm.

forkeywordslenbuilt-in functions"text"text42numbers# → 3what comes back
01

Clues and Answers

X and y

X is the clues, a table with one row per example. y is the answer for each row, a single column. Every scikit-learn model learns from this pair.

Example
X.shape  # → (20, 3)
len(y)  # → 20

The share of answers

For an answer of 0 and 1, the mean is the share of 1s: how rare the thing you are looking for is.

Example
y.mean()  # → 0.45

A baseline first

What the laziest guess scores. A model is only worth what it beats this by.

Example
always_no = (y == 0).mean()
always_no  # → 0.55
TipOn a rare event, "always no" scores very well and catches nothing. A high accuracy alone proves nothing.
02

A First Model

Make, fit, predict

Make the model, .fit() it on the clues and answers, then .predict() an answer for each row of clues.

Pattern
model = DecisionTreeClassifier(max_depth=3, random_state=0)
model.fit(X, y)
guesses = model.predict(X)
Example
model = DecisionTreeClassifier(max_depth=2, random_state=0)
model.fit(X, y)
guesses = model.predict(X)
len(guesses)  # → 20

Scoring by hand and .score()

The share of guesses that match the answers. .score() is the same sum in one call.

Example
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
by_hand = (model.predict(X) == y).mean()
by_hand == model.score(X, y)  # → True

One new row

Ask about tonight's alarm with a one-row table, the same columns in the same order.

Example
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
tonight = pd.DataFrame({"sensors": [4], "door_forced": [1], "hour": [2]})
model.predict(tonight)[0] in (0, 1)  # → True

max_depth

How many questions in a row the tree may ask. No limit, and it can learn every row by heart and still guess new ones badly.

Example
deep = DecisionTreeClassifier(random_state=0).fit(X, y)
deep.score(X, y)  # → 1.0
03

Testing Honestly

Visual reference

confusion_matrix(y_test, guesses): the real answers down the side, the guesses across the top.

guessed 0guessed 1real 0
[0][0]rightly left alone
[0][1]false alarms
real 1
[1][0]missed
[1][1]caught

Recall is caught out of every real one. Precision is caught out of every flag.

train_test_split()

Holds some rows back. The model learns on the rest and is marked on rows it never saw. The four pieces come back in this order.

Pattern
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
Example
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
len(X_test)  # → 5
TipA model marked on the rows it learned from is marking its own homework. Report the test score.

stratify=y

Keeps the share of each answer the same on both sides of the split, which matters when one answer is rare.

Example
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0, stratify=y)
abs(y_test.mean() - y.mean()) < 0.1  # → True

DummyClassifier

The baseline as a model: strategy="most_frequent" always guesses the most common answer.

Example
from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy="most_frequent").fit(X, y)
dummy.score(X, y)  # → 0.55

confusion_matrix()

A grid of right and wrong: the real answers down the side, the guesses across the top.

Example
from sklearn.metrics import confusion_matrix
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
table = confusion_matrix(y, model.predict(X))
table.shape  # → (2, 2)
04

Reading a Tree

export_text()

The tree's rules written out as text, question by question, with the column names.

Example
from sklearn.tree import export_text
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
print(export_text(model, feature_names=list(X.columns)))

feature_importances_

How much the tree leaned on each clue, adding up to 1. Put it in a Series to see the names beside the numbers.

Example
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
importance = pd.Series(model.feature_importances_, index=X.columns)
round(importance.sum(), 6)  # → 1.0
05

Predicting a Number

LinearRegression

A straight line through the data: .coef_ is how much the answer moves per unit of each clue, .intercept_ where it starts.

Example
from sklearn.linear_model import LinearRegression
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 500]})
line = LinearRegression().fit(claims[["items"]], claims["claimed"])
round(line.coef_[0])  # → 96

mean_absolute_error()

How far off the predictions are on average, in the answer's own units.

Example
from sklearn.metrics import mean_absolute_error
mean_absolute_error([100, 200, 300], [110, 190, 330])  # → 16.666666666666668

DummyRegressor

The number baseline: strategy="mean" guesses the average every time. Beat it or the model is not helping.

Example
from sklearn.dummy import DummyRegressor
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 500]})
base = DummyRegressor(strategy="mean").fit(claims[["items"]], claims["claimed"])
base.predict(claims[["items"]])[0]  # → 306.0

A residual

What happened minus what the model expected. Sort by it, biggest first, to find what the model cannot explain.

Example
from sklearn.linear_model import LinearRegression
claims = pd.DataFrame({"items": [1, 2, 3, 4, 5], "claimed": [120, 210, 290, 410, 900]})
line = LinearRegression().fit(claims[["items"]], claims["claimed"])
claims["gap"] = claims["claimed"] - line.predict(claims[["items"]])
claims.sort_values("gap", ascending=False)["items"].iloc[0]  # → 5
06

Rare Events

recall_score()

Of the real ones, the share the model caught. The number that matters when a miss is costly.

Example
from sklearn.metrics import recall_score
recall_score([1, 1, 1, 1, 0, 0], [1, 1, 0, 0, 0, 1])  # → 0.5

precision_score()

Of the ones it flagged, the share that were real. The number that matters when a false alarm is costly.

Example
from sklearn.metrics import precision_score
round(precision_score([1, 1, 1, 1, 0, 0], [1, 1, 0, 0, 0, 1]), 2)  # → 0.67
TipThe real answers go first and the guesses second. Swap them and recall and precision trade places without a word.

predict_proba()

A chance for every row instead of a yes or no. [:, 1] keeps the chance of a 1.

Example
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
chance = model.predict_proba(X)[:, 1]
len(chance)  # → 20

Moving the threshold

.predict() flags anything over a half. A lower line flags more: more caught, more false alarms.

Example
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
chance = model.predict_proba(X)[:, 1]
(chance > 0.3).sum() >= (chance > 0.5).sum()  # → True
07

Fair Scores

cross_val_score()

Five tests instead of one: the rows cut into five folds, each held back once. Report the average and look at the spread.

Pattern
cross_val_score(model, X, y, cv=5, scoring="recall")
Example
from sklearn.model_selection import cross_val_score
scores = cross_val_score(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5)
len(scores)  # → 5
spread = scores.max() - scores.min()
TipOne split is one draw. A spread as wide as the average means the single score was luck.

scoring=

What each fold is marked on: "accuracy" by default, "recall", "precision", or "average_precision" for a model whose chances you will draw a line through.

Example
from sklearn.model_selection import cross_val_score
ap = cross_val_score(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5, scoring="average_precision")
(ap >= 0).all()  # → True

HistGradientBoostingClassifier

Many small trees, each fixing the last one's mistakes. The model a data team would usually ship for tables like these.

Example
from sklearn.ensemble import HistGradientBoostingClassifier
booster = HistGradientBoostingClassifier(random_state=0).fit(X, y)
len(booster.predict(X))  # → 20

cross_val_predict()

A prediction for every row from a model that never saw that row: the fair way to give every row a chance.

Example
from sklearn.model_selection import cross_val_predict
chance = cross_val_predict(DecisionTreeClassifier(max_depth=2, random_state=0), X, y, cv=5, method="predict_proba")[:, 1]
len(chance)  # → 20
08

Words and Pipelines

pd.get_dummies()

A column of words into one True or False column per word, so a model can use it.

Example
pays = pd.DataFrame({"amount": [20, 900, 45], "merchant": ["food", "giftcards", "food"]})
list(pd.get_dummies(pays, columns=["merchant"]).columns)  # → ['amount', 'merchant_food', 'merchant_giftcards']

Leakage

A clue that is only known after the answer, or IS the answer. A score that looks too good usually is.

Example
leaky = alarms.assign(police_called=alarms["real"])
(leaky["police_called"] == leaky["real"]).mean()  # → 1.0
TipAsk of every column: would I have this at the moment the prediction is made?

A time split

Train on the past and test on what came after, the way the model will really be used.

Example
past = alarms.index < 15
X_past, y_past = X[past], y[past]
X_future, y_future = X[~past], y[~past]
len(X_future)  # → 5

OneHotEncoder and make_pipeline

An encoder that remembers the words it learned, and a pipeline that carries the preparation and the model as one object, so a raw row can go straight in.

Example
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import make_pipeline
pays = pd.DataFrame({"amount": [20, 900, 45, 700], "merchant": ["food", "giftcards", "food", "giftcards"], "fraud": [0, 1, 0, 1]})
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["merchant"]), remainder="passthrough")
pipe = make_pipeline(prep, DecisionTreeClassifier(random_state=0))
pipe.fit(pays[["amount", "merchant"]], pays["fraud"])
pipe.predict(pd.DataFrame({"amount": [850], "merchant": ["giftcards"]}))[0]  # → 1
09

Explaining and Auditing

permutation_importance()

Shuffles one clue at a time and measures how far the score falls. A big fall means the model leans on that clue.

Example
from sklearn.inspection import permutation_importance
model = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
result = permutation_importance(model, X, y, n_repeats=5, random_state=0)
len(result.importances_mean)  # → 3

False alarms by group

Among the honest rows only, the share flagged in each group. A gap means the model is harder on some people than others.

Example
audit = pd.DataFrame({"district": ["A", "A", "B", "B"], "fraud": [0, 0, 0, 0], "flag": [True, False, False, False]})
honest = audit[audit["fraud"] == 0]
honest.groupby("district")["flag"].mean()["A"]  # → 0.5

IsolationForest

Finds odd rows with no answers at all. score_samples gives every row a score, lower for the odder ones.

Example
from sklearn.ensemble import IsolationForest
claims = pd.DataFrame({"amount": [40, 42, 38, 41, 39, 400]})
finder = IsolationForest(random_state=0).fit(claims)
claims["score"] = finder.score_samples(claims)
claims.sort_values("score")["amount"].iloc[0]  # → 400
10

Which Score, When

The questionThe scoreCall
How often is it right?Accuracymodel.score(X_test, y_test)
Of the real ones, how many were caught?Recallrecall_score(y_test, guesses)
Of the flags, how many were real?Precisionprecision_score(y_test, guesses)
How good are its chances, at every line?Average precisionscoring="average_precision"
How far off is a number?Mean absolute errormean_absolute_error(y_test, predicted)

Every score here is worth what it beats the baseline by. Score the dummy first.

Scikit-learn Cheat Sheet · QueryCaseThe full course is at querycase.com