A first honest model on a new project: OLS or logistic regression, a frozen split, a naive baseline and a metric you can defend

Prove the method on the simplest model first

A simple model on a clean split tells you something true about your data, while a powerful model on a flawed split only tells you how well it learnt the flaw.

The order matters. Teams that open with gradient boosting tend to discover, weeks later, that the split leaked or that the metric answered a question nobody asked. A linear model is fast to fit, easy to inspect and hard to flatter, so its errors point at the data and the setup rather than at tuning. If the plumbing is right for OLS or a logistic regression, adding a forest later costs one more line in the comparison; if it is wrong, no model will rescue it.

The starting material is the scope document written in the first milestone: the decision the model informs, the question, the target, two metrics and a guard-rail on what the model must never decide alone. Everything below is derived from it. The common thread of the course, a credit default score built in the baseline milestone, already taught the three habits that carry over: a frozen split, a naive reference and a decision threshold. The second milestone moves them onto the team's own data.

The six pieces of a first honest model

1. A frozen split. Training, validation and a test set that stays sealed until the final delivery, with nothing leaking between them.
2. A naive reference. Computed on the training rows alone, it ignores every feature.
3. One learnt model. OLS for a continuous target, logistic regression for a binary one.
4. A justified metric. One sentence ties it to the decision recorded in the scope document.
5. Validation results and limits. The model against the naive reference, read honestly, and where it stops holding.
6. Reproducibility. One command rebuilds every number from a fresh clone of the repository.

Back to contents.

The target chooses the model, the baseline and the metric

Whether you fit a linear or a logistic regression is not a preference: it follows from what the target is, and the naive reference and the metric follow from it too.

Read the target in the scope document and the rest falls out. A continuous target, an amount, a duration, a count large enough to treat as a quantity, calls for ordinary least squares, covered in least squares and linear regression. A binary target calls for a logistic regression fitted by maximum likelihood, covered in maximum likelihood and logistic regression.

For a regression, express the metric in the unit of the target, because a decision-maker can act on "we are off by about two hours per repair" and cannot act on an $R^2$. For a binary target, the decision usually turns on a probability and a cost, and the threshold that minimises the expected cost is the subject of the decision threshold note. Targets with several classes, an ordering or a strong time dimension follow the same principle, but the modelling choice and its reason go into the decision log before the first computation, not after the results.

Back to contents.

A frozen split that cannot leak

Cut the data before computing anything, learn every preprocessing step from the training rows only, and record the cut so that nobody can quietly redo it.

Split first, before any mean, any imputation, any plot that informs a choice. Three parts: the training set, where the model learns; the validation set, where you measure and decide; the test set, which stays closed until the final delivery. When rows carry a date, cut by date so that the model learns from the past and is scored on the future, as it will be used. When the same customer, patient or machine appears on several rows, cut by group so that it never sits on both sides.

Then anything that learns from data, imputation, scaling, encoding, feature selection, is fitted on the training rows alone. The simplest way to guarantee it is a scikit-learn Pipeline, backed by a unit test that checks, for instance, that the mean stored by the scaler is the training mean. Finally freeze the split: a hash of the raw file, one seed kept in config.yaml, and the list of identifiers in each part, all committed. The sketch below writes the three parts once, then fits a logistic regression that never sees the test file.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# split.py, run once: cut by date, write three files, log their hashes
df = pd.read_csv("data/raw.csv", parse_dates=["decided_on"])
parts = {"train": df.decided_on < "2024-01-01",
         "valid": (df.decided_on >= "2024-01-01") & (df.decided_on < "2024-07-01"),
         "test": df.decided_on >= "2024-07-01"}
for name, mask in parts.items():
    df[mask].to_csv(f"data/{name}.csv", index=False)

# fit.py: reads train and valid only, every statistic is learnt on train
train, valid = pd.read_csv("data/train.csv"), pd.read_csv("data/valid.csv")
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), NUM),
    ("cat", OneHotEncoder(handle_unknown="ignore"), CAT),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
model.fit(train[NUM + CAT], train["y"])

Back to contents.

A naive reference and a metric you can defend

Every result needs a reference that uses no feature at all, and a metric chosen because of the decision, otherwise a good-looking number may mean nothing.

The naive reference looks at no variable. For a regression it predicts the training mean of the target for every row, or the median if the metric is the mean absolute error, since the median is what minimises it. For a binary target it predicts the training base rate $\pi$ for everyone. Its scores are the numbers to beat, and they are often harder to beat than they look.

A worked example with a base rate of 0.20

1. Brier score. The naive reference scores $0.20 \times 0.80 = 0.16$, so a model whose Brier score is above that does worse than simply knowing the base rate.
2. Log-loss. $-(0.20 \ln 0.20 + 0.80 \ln 0.80) \approx 0.500$, the figure a logistic regression has to bring down.
3. Accuracy. Answering no for everyone is right 80 % of the time, which is why accuracy alone flatters a useless model when classes are imbalanced.

The metric is then justified in one sentence that starts from the decision in the scope document, never from habit. For instance: "we report log-loss because the maintenance planners act on a probability, and the expected cost at the chosen threshold because a missed failure costs far more than an unnecessary inspection". Accuracy rarely survives that sentence when classes are imbalanced. For a regression, RMSE fits when large errors hurt disproportionately, MAE when every unit of error costs the same.

Back to contents.

Results on validation, with the test still sealed

Report the model against the naive reference on the validation set, with an idea of the uncertainty, and keep the test set closed for the final verdict.

The core of the deliverable is a small table, and every number in it is written by the code into a results file such as metrics.json, never typed by hand. At least two rows, the naive reference and the learnt model, and a column for the gap between them. Everything is measured on the validation set: training scores are optimistic by construction, and test scores, once looked at, can no longer serve as an unbiased verdict.

import numpy as np
from sklearn.metrics import brier_score_loss, log_loss

y = valid["y"].to_numpy()
p_naive = np.full(len(y), train["y"].mean())   # training base rate
p_model = model.predict_proba(valid[NUM + CAT])[:, 1]

def gap(i):  # log-loss of the model minus log-loss of the naive reference
    return log_loss(y[i], p_model[i], labels=[0, 1]) - log_loss(y[i], p_naive[i], labels=[0, 1])

rng = np.random.default_rng(SEED)              # the one seed from config.yaml
boot = [gap(rng.integers(0, len(y), len(y))) for _ in range(2000)]
print(f"Brier naive {brier_score_loss(y, p_naive):.3f}, model {brier_score_loss(y, p_model):.3f}")
print(f"gap {gap(np.arange(len(y))):.3f}, 95% interval {np.percentile(boot, [2.5, 97.5]).round(3)}")

The interval on the gap comes from a bootstrap of the validation rows; with several time-ordered folds, the spread across folds does the same job. It answers the first question a reader will ask: is the improvement larger than what resampling alone would produce? A gap of $-0.045$ whose interval stays well below zero is a result; a gap of $-0.004$ with an interval straddling zero is not yet one.

Reading the coefficients

A linear model earns its place partly because it can be read. In OLS, $\hat\beta_j$ is the change in the predicted target, in its own unit, when $x_j$ rises by one unit and the other features stay fixed; if the pipeline standardises inputs, the unit is one standard deviation, and the write-up should say so. In a logistic regression, $\beta_j$ moves the log-odds, and $e^{\beta_j}$ is the odds ratio: a coefficient of 0.4 per standard deviation multiplies the odds by about 1.49. Two cautions. Correlated features share credit, so a sign can flip when a neighbour is added. And a coefficient describes an association in this data, not the effect of intervening on the feature.

Back to contents.

Written limits and a one-command rebuild

Limits are specific statements about where the model does not hold, and reproducibility means another team gets your exact numbers from a fresh clone with one command.

Limits are not a courtesy paragraph. Five precise lines are enough when each one carries information a decision-maker can use.

• Population and period. Who and when the data describes, hence for whom the model says nothing.
• The target. What it measures badly or leaves out.
• Excluded features. What was set aside for fear of leakage or on principle, and what that costs.
• Validation size. Whether a gap of 0.01 can be told apart from noise with this many rows.
• Forbidden uses. What the model must never decide alone, taken from the guard-rails of the scope document.

Reproducibility is a condition rather than a bonus: another team clones the repository, runs make reproduce inside the dev container and obtains the same metrics.json, with continuous integration green on the last commit. If the data cannot be published, the README explains how an auditor can obtain it, or the repository ships an anonymised sample that runs the pipeline end to end. A reader then finds, in a predictable order, a short written note (split and leakage, naive reference, metric, results table, limits), the code and its tests, the results file, the scope document updated if the problem has moved, and the decision log with an appendix on how AI tools were used.

Eight checks before handing over

1. Split. Hashed data, a single seed, the identifiers of each part committed.
2. Pipeline. No preprocessing step is fitted outside it.
3. Sealed test. A search of the code shows that no script of this step reads the test file.
4. Naive reference. Its row is in the results file, next to the model.
5. Metric. Justified in one sentence that starts from the decision.
6. Limits. Five of them, each specific enough to act on.
7. Fresh clone. The rebuild runs from a temporary folder, not from the author's laptop.
8. Ownership. Each part has a named owner who can explain it.

Back to contents.

Optional practice: rehearsing the champion comparison

On a public credit default dataset, you can already rehearse the comparison of model families that the final delivery will require, with nothing at stake.

The course keeps the former, more ambitious brief as optional practice on the common thread, the UCI Default of Credit Card Clients data (Yeh and Lien, 2009) presented in the credit default scoring project. It is the dress rehearsal for the model comparison that the final delivery asks for on the team's own data, and a safe place to make mistakes.

• One comparison command: an L1 logistic regression as reference, a random forest, gradient boosting and a kernel model, over three seeds and five folds, hyperparameters tuned by nested cross-validation, the test set left out.
• Corrected intervals: the folds overlap, so the plain variance of the fold scores is too small, and the Nadeau and Bengio correction below inflates it.
• A champion rule written before the results, filed as a timestamped issue, so that the criterion cannot be picked to suit the table.
• A model card with a fairness audit that goes beyond dropping the sensitive column: a proxy test, then group metrics with intervals.
• Three algorithms written by hand, gradient boosting, kernel ridge and a one-dimensional EM, each checked against scikit-learn by an automated test.

With $J = 15$ measurements and scoring folds a quarter the size of the training part, the factor is $1/15 + 1/4$ instead of $1/15$, so the variance is multiplied by 4.75. The verdict is then read on the paired difference between a challenger and the reference: if its corrected interval contains zero, the gain is not established and the simpler model stays. The methods behind this practice are covered in random forests and gradient boosting and SHAP, fairness, PCA and clustering.

Back to contents.

Key takeaways

1. Prove the method on a simple model first: a linear model on a clean split is worth more than a boosted one on a split that leaks.
2. The target decides: OLS for a quantity, logistic regression for a yes or no, and the naive reference and the metric follow from the same choice.
3. Split before computing anything, fit every preprocessing step on the training rows inside a pipeline, and freeze the split with a hash, a seed and the identifiers.
4. A naive reference that ignores every feature is the first row of the results table; with a base rate of 0.20 its Brier score is already 0.16.
5. Justify the metric in one sentence that starts from the decision, measure on validation with an interval on the gap, and keep the test sealed.
6. Five specific limits and a one-command rebuild from a fresh clone turn a number into a result that someone else can trust.

Glossary

ordinary least squares (OLS)

Linear regression whose coefficients minimise the sum of squared residuals.

logistic regression

Model of the probability of a binary outcome through a linear function of the log-odds, fitted by maximum likelihood.

naive baseline

A reference prediction that ignores every feature: the training mean, the median or the base rate.

data leakage

Information used by the model that would not be available at decision time, or that comes from the rows used for evaluation.

frozen split

A train, validation and test partition fixed once, recorded with a data hash, a seed and identifiers, and never redone to improve results.

validation set

Rows used to measure models and make choices, distinct from the training rows.

test set

Rows kept sealed until the final verdict and used once.

Brier score

Mean squared difference between a predicted probability and the observed 0 or 1 outcome.

log-loss

Mean negative log-likelihood of the observed outcomes under the predicted probabilities.

odds ratio

Factor by which the odds p/(1 − p) are multiplied when a feature rises by one unit; the exponential of a logistic coefficient.

bootstrap

Resampling rows with replacement to estimate how much a statistic would vary.

reproducibility

Another person obtains the same numbers from the same code and data with a single command.

scope document

The written statement of the decision, target, metrics and guard-rails on which a project is built.

decision log

Dated entries recording each choice, the alternative rejected and the evidence behind it.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: SHAP, fairness, PCA and clusteringNext: Neural networks, drift, conformal prediction

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkFraming an ML project: the scope document
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA simple model with a complete method
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.