Framing a machine learning project before any code: the question, the decision, the target and a six-line scope document

The question, in one sentence

A useful question names a decision, a way to measure success, a scope and the person who will act on the answer, all in one sentence.

"Use AI for logistics" is an ambition, not a question: nobody can tell whether it has been answered. The framing method of the course, introduced in its first session (frame before you model), asks for a sentence with four markers. A decision: something that will be done differently depending on the answer. A measure: how we will know the decision improved. A scope: which population, which place, which period. An owner: a role that will actually act on the prediction.

On the course's common thread, the owner was a credit analyst who reviews applications; in the fictional example used throughout the brief, it is the manager of a delivery depot who decides, each evening, which customers to call so that tomorrow's drop-offs go smoothly. If the question needs two sentences, it usually hides two decisions, and the team should pick one.

Back to contents.

The decision and what each error costs

Write who decides, what the real options are and when the choice is made, then put a price on acting for nothing and on failing to act.

The decision comes first because it fixes everything that follows. Name a role rather than a person (an operations manager, a credit analyst, a buyer, an emergency dispatcher) and the moment at which that role acts, because that moment decides which information the model may use. Then list two or three concrete options that the prediction can tip one way or the other.

Every decision has two ways of going wrong: acting when it was not needed, and not acting when it was. They almost never cost the same. Writing both prices down, even roughly, is what later lets you choose a threshold instead of accepting 0.5 by default. "Understanding delays better" is not a decision, since nobody behaves differently whatever the answer; a project without an identifiable decision-maker tends to end as a demo with nothing at stake.

• Who: a role and the deadline by which it acts.
• What: the real options, the ones the prediction can change.
• False alarm: the cost of acting for nothing (a wasted call, a refused good client).
• Miss: the cost of not acting in time (a complaint, a default accepted).

Back to contents.

A dated target, and the leakage it rules out

The target is a column recorded by a rule at a known date, and anything that happens after the moment of prediction is forbidden as an input.

A target is not the phenomenon itself but a dated trace of it, recorded according to a rule. The scope document gives four facts: the exact column name and how it is computed, the horizon of the prediction, and the date at which the true value is known. For the delivery example: late equals one when a parcel arrives after the promised slot, predicted the evening before, known on delivery.

Those dates are the main protection against data leakage. A target that is already known before the decision is suspicious, and any variable recorded after the moment of prediction must not be an input: in the example, nothing that happens during the next day's round. The decision threshold and reproducible ML note shows how a leakage map, column by column, makes this check systematic.

Add one sentence on what the column does not capture. In the example, an absent customer is counted as late, and one minute late weighs as much as a day. This is not modesty: it stops the team from promising the decision-maker more than the column actually measures.

Back to contents.

A metric that measures the decision, and safeguards you can check

Judge the model in the units the decision-maker reads, set the level at which it becomes useful, and list what it must never do even if it would score better.

The metric that matters is the one that says whether the decision got better: an expected cost in euros, hours saved, delays avoided for a given number of calls. Write the usefulness threshold too, the level at which the model beats doing nothing, or beats the simple rule already in place. Then name the model metric that tracks it on the validation set: log-loss or Brier score for probabilities, PR-AUC for a rare class, mean absolute error for a quantity.

Safeguards state what the model must never do. They come in three families: forbidden variables (a protected attribute, directly identifying personal data), people to protect (a group that an error would hurt more than others) and leakage (anything known after the decision). On the credit thread, sex was excluded while marital status and age were audited. A safeguard is worth writing only if it can be verified in the code or in the data, and it is more credible when a named role signs it and has the power to stop the model.

Back to contents.

Data you can open today, a plan with three models, and the team

Name a source you can open now with its licence, period and gaps, plan a naive, a better and a heavy model, and say who is on the team.

The data line sinks more projects than any other, because teams fill it in last. It needs an address that works today, not "we will ask the company"; a licence that has been read and suits a public student project; the period covered and the number of rows, which tells you whether a split by date is possible; and an honest list of what is missing: the column you wish you had, gaps, known collection biases. If access depends on a third party, write when you expect it and name an equivalent public fallback. The open data projects note lists sources and shows how to read a licence quickly. A line you cannot fill in this week is a reason to change the topic, not the model.

The plan, in three models

1. Naive. The reference to beat: a historical rate, an average, or the business rule already in use.
2. Better. The best reasonable model, often a regularised linear or logistic regression.
3. Heavy. A more complex model, such as gradient boosting, that tests whether complexity actually pays.

The plan also says what should exist at each later milestone and who does what, with three roles (data, model, review) that rotate. It is not a binding commitment: the plan will change, and the file history will show how. The brief ends the template with who we are, in two parts. First the persona: the company the team plays for the project, whether a consultancy working for a bank, the data unit inside a bank, a fintech's risk function or the quants of a credit fund, with its client or counterpart and the person who signs off. Then a short team profile of the real people, one line per member written by that member: background, the job they are aiming for (an honest "not sure yet" is fine), why they chose this track and their interests. The persona tells everyone on whose behalf the team speaks; the profiles are used to assign roles, to ask each person questions close to their future job, and to connect the project with their applications.

Back to contents.

Two worked examples: a parcel-delivery agency and credit risk

A fictional delivery team shows a complete file and how two error prices give a threshold; a file filled in on public credit data shows the same template on a finance topic.

The brief fills the whole template for an imaginary team, with imaginary figures: read the form, not the content. The manager of one depot decides each evening, before a fixed cut-off, whether to have a customer phoned. A wasted call is priced at 8 euros and an unannounced delay at 25 euros, because it triggers a complaint. The target is dated after the decision, the metric is an expected cost per evening, the safeguards forbid refusing a delivery and using any personal data of the recipient, and the data line names an anonymised export of past rounds, a written access request and a public fallback with its licence, plus the missing weather, to be joined by date and town.

In words, with the example's prices (illustrative figures)

1. Two prices. A false positive is a call for nothing ($c_{FP} = 8$), a false negative is a missed delay ($c_{FN} = 25$), and a correct decision is counted as free.
2. Compare. Calling a parcel with delay probability $p$ costs $8(1-p)$ on average and not calling costs $25p$, so calling is worth it when $25p > 8(1-p)$.
3. Threshold. That happens when $p$ exceeds about 0.24, provided the probabilities are calibrated, which is why the metric line also names log-loss.
4. Naive reference. With 4 % late parcels, never calling costs $0.04 \times 25 = 1$ euro per parcel and calling everyone about 7.7 euros ($0.96 \times 8$), so the model is useful only below 1 euro.

Credit risk on Home Credit Default Risk

The brief also fills the template on a real dataset: Home Credit Default Risk, a 2018 Kaggle competition built on data from a consumer lender whose clients often have a thin credit history or none at all. The persona is invented: Aurore Data Conseil, a data consulting firm working for the lender's risk department, whose director signs off the threshold. The question asks which of the month's applications a credit officer should route to a manual review, so as to cut defaults while turning away no more good clients than today.

• Before handing in, check that the question carries its four markers, the decision has a role and both error prices, the target has a horizon and a date of knowledge, the metric has a usefulness threshold, each safeguard can be verified and the data line has a source, a licence, a period and its gaps.
• Check that the plan, the persona and the team profile are written, and that the file is pushed to the main branch of the shared repository.
• The neighbour test: someone from another team reads the file without any explanation and restates the question, the decision, the target and the metric. Any hesitation means that line needs rewriting.

Back to contents.

Practice: proving a baseline on the common thread

On public credit data, a team can rehearse the method the next milestone will apply to its own project: a reference, a note, an honest ablation.

The second part of the brief is recommended practice on the credit default common thread (UCI Default of Credit Card Clients, Yeh and Lien, 2009). It rehearses what the next milestone asks on the team's own data. A baseline is the simple, honest model that any candidate must beat by a measured margin. The first reference is the constant model that predicts the base rate $\pi$, about 22.1 % defaults, for every client; its scores can be computed by hand.

Six pieces, in the order to open them

1. A rerunnable repository. Code, tests, a lockfile, continuous integration and one command that goes from the raw CSV to a metrics file.
2. Two models by hand. Least squares by the normal equations and by SVD (least squares), logistic regression by gradient descent (logistic regression), each compared with scikit-learn, gap printed.
3. A two-page note. Page one: framing, data, frozen split, leakage map. Page two: the model against the constant, calibration with a reliability curve, the cost-based threshold and what happens if costs move by a fifth.
4. A pre-registered ablation. Predict what is lost without each group of columns, file the prediction as a timestamped issue, compute on validation with the test set sealed, comment each gap.
5. A decision log. Dated entries with the rejected alternative and the command that proves the figure.
6. An AI appendix. Each use with its level, and two assistant claims proved wrong by a failing test, a computation or a figure.

The note copies the outline a bank's model committee reads: an independent validation team reruns the figures from the repository, then looks for the leak, the unjustified threshold, the uncalibrated probability, before the committee approves, approves with conditions or refuses. Every figure in the note must come from the metrics file. Four pitfalls recur: a scaler fitted on the whole CSV before the split (a test compares its stored mean with the training fold), an AUC computed on hard labels instead of probabilities, random seeds scattered across notebooks instead of one configuration file, and a README that only runs on its author's machine, caught only when someone else clones the repository and follows it.

Back to contents.

Key takeaways

1. Write the question in one sentence with a decision, a measure, a scope and an owner; if it needs two sentences, pick one decision.
2. Name who decides, when, between which options, and price both errors: acting for nothing and failing to act.
3. Date the target and the moment of prediction: anything recorded after that moment is leakage and cannot be an input.
4. Measure the decision in the decision-maker's units, with a usefulness threshold, and never rely on accuracy alone for a rare class.
5. Write safeguards that can be checked, a data source you can open today with its licence and gaps, and a plan with a naive, a better and a heavy model.
6. Prove a baseline with a constant reference, models written by hand, a traceable note and an ablation whose predictions were filed first.
7. Say who you are: the company the team plays, its client and who signs off, then the real people behind it.

Glossary

Scope document

A short file that states the question, decision, target, metric, safeguards, data and plan of a project.

Problem statement

The question a project answers, written so that someone can act on the answer.

Decision owner

The role that acts on the prediction, at a known moment.

Expected cost

The average cost of the decisions taken with a model, weighting each error by its price.

Usefulness threshold

The level at which a model becomes better than doing nothing or than the existing rule.

Prediction horizon

The time between the moment of prediction and the event predicted.

Data leakage

Information unavailable at decision time that reaches the model and inflates its scores.

Safeguard

Something the model must never do, stated precisely enough to be verified.

Team persona

The company a project team plays for its project, with its client and the person who signs off.

Accuracy

The share of correct predictions; misleading when one class is rare.

Baseline

The simple reference model any candidate must beat by a measured margin.

Ablation study

Removing groups of inputs one at a time to measure what each contributes.

Pre-registration

Writing predictions down, timestamped, before running the experiment.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Decision threshold and reproducible MLNext: Kernels, SVM and decision trees

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkFraming an ML project: the scope document
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA simple model with a complete method
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.