Logistic regression, lift charts and market basket analysis

The spring campaign that pays for itself, or does not

Every March The Gift Bow, my Solidus demo hamper shop at hampers.keferboeck.com, has the same conversation with itself. Christmas is over and there are about 20,000 people on the customer file who have bought at least once. Easter and Mother's Day are coming, and a printed spring brochure with a code inside costs roughly £1.20 per household once you add print, postage and the discount that gets redeemed. Mail all 20,000 and you spend £24,000 before a single hamper leaves the warehouse. Mail nobody and you leave revenue on the table. Mail "the good customers" by gut feeling and you never find out what you missed.

The question is not "should we mail". It is "who, and how deep". That is a yes or no outcome per customer: respond or not. This chapter is about the most common outcome in marketing, the binary one, and about the tool built for it: logistic regression. I will use it twice. First to rank customers for the spring campaign and find the depth where the marginal brochure stops paying for itself. Then on the order lines, to answer a basket question: given the wine hamper in the cart, how likely is the cheese hamper, and what should the shop recommend. The numbers throughout are illustrative; the method is exactly what I run for clients.

Where this sits in the series

This is chapter 7 of 21 and the fourth chapter of part two, dependent variable techniques (chapters 4 to 10). The previous chapter, Poisson regression: modelling how many times customers do things, handled counts. The next one, Survival analysis, churn and customer lifetime value, adds time to the picture: not whether a customer leaves, but when. Logistic regression sits between them as the workhorse: half of what gets sold as "predictive marketing" is a logistic model with a decent feature table behind it.

Why ordinary regression breaks on a yes or no outcome

Code response as 1 and no response as 0, fit an ordinary least squares line, and three things go wrong. The prediction is unbounded, so a very engaged customer gets a predicted response of 1.3, which is not a probability. The errors cannot be normally distributed, because the outcome only takes two values. And every effect is forced to be linear: moving from 1 to 2 orders adds as much probability as moving from 9 to 10, which nobody believes.

Logistic regression fixes all three with one idea. Instead of modelling the probability directly, it models the log of the odds. Odds are the probability of the event divided by the probability of the non event: a 20 percent response rate is odds of 0.25, 50 percent is odds of 1, 90 percent is odds of 9. Odds run from zero to infinity, and their natural log, the logit, runs from minus infinity to plus infinity, which is exactly what a linear equation wants to produce.

$\ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_k x_k$

Here $p$ is the probability that the customer responds, $x_1$ to $x_k$ are the things we know about the customer, $\beta_0$ is the intercept, the log odds for a customer with every $x$ at zero, and each $\beta_j$ is the change in the log odds for one unit more of $x_j$ holding the rest constant.

Solve that for $p$ and you get the logistic function, the S shaped curve that gives the method its name:

$p = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_k x_k)}}$

The symbol $e$ is Euler's number, about 2.718. Whatever the linear part produces, the function squashes it between 0 and 1: a score of 0 becomes a probability of 0.5, a score of 2 becomes 0.88, a score of minus 4 becomes 0.018. The curve is steepest in the middle and flat at both ends, which is a fair description of people: pushing an already loyal customer harder does very little.

The logistic function: a linear score on the horizontal axis becomes a probability between 0 and 1.

The betas are estimated by maximum likelihood, not least squares, so there is no R squared in the usual sense; the diagnostics that matter are different, and I come to them below.

Reading coefficients as odds ratios

The betas live on the log odds scale, which nobody thinks in. Exponentiate them and you get odds ratios, which people can actually use:

$\text{OR}_j = e^{\beta_j}$

An odds ratio of 1.73 for the gift buyer flag means that a gift buyer has 1.73 times the odds of responding compared with a self buyer with otherwise identical characteristics. Odds, not probability. If the self buyer's probability is 3 percent (odds 0.031), the gift buyer's odds are 0.054 and their probability about 5.1 percent.

Here is the spring response model for The Gift Bow, fitted on last year's spring campaign (20,000 mailed, 788 responded, so a 3.9 percent base rate). Every variable is measured as of the day the brochure went out.

Variable	Coefficient	Std. error	Odds ratio	VIF
Intercept	minus 3.40	0.21
Months since last order	minus 0.09	0.012	0.91	1.6
Orders in the last 24 months	0.28	0.04	1.32	2.1
Gift buyer (1 = yes)	0.55	0.11	1.73	1.2
Corporate account (1 = yes)	0.42	0.16	1.52	1.3
Opened last spring's email (1 = yes)	0.78	0.10	2.18	1.4
Average order value, per £10	0.06	0.02	1.06	1.9

Reading it line by line. Each month since the last order multiplies the odds of responding by 0.91, so a customer who last bought 12 months ago has about 0.91 to the power of 12, roughly 0.32 times the odds of someone who bought last month. Each extra order in two years multiplies the odds by 1.32. Gift buyers have 73 percent higher odds than self buyers; that is the segment the spring brochure is really for. Corporate accounts respond better than private ones even after controlling for order count, presumably because offices buy Easter hampers for staff. Opening last year's spring email is the strongest single signal, an odds ratio above 2. Average order value matters, but modestly: £50 more in the basket raises the odds by about 34 percent.

Rule of thumb for the standard errors: a coefficient below about twice its standard error could plausibly be zero. Everything here clears that bar; the corporate flag only just.

The last column, VIF, is the variance inflation factor, and it is your collinearity alarm:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

$R_j^2$ is the R squared you would get by regressing variable $j$ on all the other predictors. If they explain 80 percent of the variation in $x_j$, its VIF is 5, and its coefficient is being estimated on the 20 percent of information that is genuinely its own. I get nervous above 5 and remove or combine variables above 10. Order count and order value are mildly related here (frequent buyers also buy bigger), hence 2.1 and 1.9; nothing alarming.

The classic symptom of collinearity is a sign flip: a variable that should obviously increase response comes out negative, because it is fighting a near twin for the same information. If your model says frequent buyers respond less, do not publish a clever explanation. Check the VIFs, drop one of the twins and refit.

From probabilities to a ranked list

Scoring is arithmetic: compute the linear part for every customer on the file, push it through the logistic function, sort the 20,000 people from most to least likely and cut them into ten equal groups of 2,000: deciles. Decile 1 is your best 10 percent, decile 10 your weakest.

The scoring pipeline. The loop at the end is the point: every campaign feeds the next model.

Now the campaign economics. Each brochure costs $c$, about £1.20 here. Each response is worth $m$ in contribution margin, the revenue minus product cost and fulfilment, roughly £38 on an average spring order of £68. For a decile with $n_d$ customers and response rate $r_d$, the profit from mailing it is

$\pi_d = n_d \left( r_d \, m - c \right)$

and it is positive whenever the decile's response rate is above the breakeven rate

$r^* = \frac{c}{m}$

which is 1.20 divided by 38, about 3.2 percent. Any decile that responds above 3.2 percent pays for itself; any decile below it loses money on every brochure. That is the whole "how deep" decision in one line.

The lift of a decile is its response rate relative to the average:

$\text{Lift}_d = \frac{r_d}{\bar{r}}$

where $\bar{r}$ is the overall response rate, 3.9 percent. A lift of 3 in the top decile means those customers respond three times as often as a random pick. Cumulative lift does the same for the top $k$ deciles together, and that is what the gains chart plots.

The decile table, and where marginal profit hits zero

Here is the scored file for this spring's campaign, with last year's response rates applied decile by decile.

Decile	Customers	Response rate	Responders	Revenue £	Decile profit £	Cumulative profit £	Cumulative lift
1	2,000	12.5%	250	17,000	7,100	7,100	3.17
2	2,000	8.0%	160	10,880	3,680	10,780	2.60
3	2,000	5.5%	110	7,480	1,780	12,560	2.20
4	2,000	4.0%	80	5,440	640	13,200	1.90
5	2,000	3.0%	60	4,080	minus 120	13,080	1.68
6	2,000	2.2%	44	2,992	minus 728	12,352	1.49
7	2,000	1.6%	32	2,176	minus 1,184	11,168	1.33
8	2,000	1.2%	24	1,632	minus 1,488	9,680	1.21
9	2,000	0.8%	16	1,088	minus 1,792	7,888	1.09
10	2,000	0.6%	12	816	minus 1,944	5,944	1.00

Read down the table. Decile 1 responds at 12.5 percent, more than three times the average; 250 orders, £17,000 of revenue, and after £2,400 of brochures a profit of £7,100. By decile 4 the response rate is 4.0 percent, just above the 3.2 percent breakeven, and the decile adds £640. Decile 5 responds at 3.0 percent, just below breakeven, and loses £120. From there on every additional decile destroys money, and mailing the entire file ends at £5,944 of profit instead of the £13,200 you reach by stopping after decile 4.

So the answer is: mail 8,000 people, not 20,000. That saves £14,400 in brochures, gives up 188 orders from deciles 5 to 10, and more than doubles the campaign profit. The revenue column is what the marketing team wants to look at; the cumulative profit column is what the owner should look at. They peak in different places, and that gap is where most campaign budgets are quietly wasted.

One refinement matters. The breakeven depth is a point estimate. With 2,000 people per decile the response rate in decile 4 carries a standard error of about 0.45 percentage points, so "4.0 percent" really means "somewhere between 3.1 and 4.9". I tell clients that decile 4 is a coin flip and decile 5 is a no, which is a more honest sentence than "mail exactly 8,000". And because print gets less expensive per unit at volume, I recompute $c$ and $m$ per decile rather than assuming constants.

Gains chart: what a good ranking looks like

The gains chart plots the share of all responders you capture against the share of the file you mail. Random selection is the diagonal: mail 30 percent of the file, catch 30 percent of the responders. The model's curve should bow above it.

Cumulative gains for the spring model against random selection. Mailing the top 40 percent of the file captures 76 percent of all responders.

The height of the curve above the diagonal in the first deciles is where the money is; where it goes flat you are mailing people who will not respond regardless. And the gap between the training curve and the curve on a holdout sample is your honesty check: if the holdout curve is noticeably lower, the model has memorised last year rather than learned something about customers.

The bar chart of cumulative profit says the same thing in pounds:

Cumulative campaign profit by mailing depth. The peak at decile 4 is the answer to "how deep".

Calibration, imbalance and when a tree is better

Ranking is one thing; the actual probabilities are another. A model is well calibrated when the customers it scores at 10 percent really do respond at about 10 percent. Logistic regression is usually well calibrated on its training data, but it drifts: this year's spring is not last year's. I check by binning the predicted probabilities into ten groups and comparing predicted with observed response in each. For the depth decision calibration matters more than for ranking, because the breakeven rule compares an absolute probability against $c / m$. If the model is systematically 20 percent too optimistic, your breakeven decile moves.

Class imbalance is the other worry, usually overdone. With a 3.9 percent response rate, 96 percent of the rows are zeros. Logistic regression copes as long as you have enough ones in absolute terms, and 788 responders is plenty for seven variables. Do not rebalance the training data by throwing away zeros or duplicating ones and then read the predicted probabilities as real: that destroys calibration, and you end up mailing to a breakeven computed in a fictional world. If you must downsample for speed, correct the intercept afterwards.

Tree based models, gradient boosted trees in particular, will often beat logistic regression by a few points of lift, because they find interactions on their own. But logistic regression has three advantages that matter more often than the extra lift. You can read it: the odds ratio table above is a management conversation in itself. It is stable on small files, and 20,000 customers with 788 responders is small for boosting. And it degrades gracefully; when the world shifts, a logistic model gets a bit worse, while a tree can fall off a cliff. My rule: start logistic, add two or three interaction terms by hand where the business logic suggests them, and reach for trees only when the holdout gains chart says the complexity is being paid for.

Market basket analysis with the same tool

Market basket analysis asks which products go together, and the textbook approach is association rules: how often wine and cheese share a basket (support), how often cheese appears given wine (confidence), and how much more often than chance (lift). Their weakness is that they consider one antecedent at a time. The wine hamper might look like it drives cheese purchases when both are really bought by corporate accounts assembling gift sets, and the driver is the account type.

Logistic regression handles this by making the presence of one product the outcome, and everything else in the basket, plus what you know about the buyer, the predictors:

$P(\text{cheese} \mid \text{basket}) = \frac{1}{1 + e^{-(\beta_0 + \beta_{\text{wine}} x_{\text{wine}} + \beta_{\text{choc}} x_{\text{choc}} + \beta_{\text{corp}} x_{\text{corp}} + \beta_{n} x_{n})}}$

Each $x$ is a 1 or 0 flag for wine hamper, chocolate hamper and corporate buyer, $x_n$ counts the other items already in the cart, and the outcome is whether the cheese hamper is in the same order. Here is the model on 14,000 Gift Bow orders (the cheese hamper appears in 9 percent of them):

Predictor	Coefficient	Odds ratio	Reading
Intercept	minus 2.55		base odds of about 0.078
Wine hamper in basket	1.10	3.00	triples the odds of cheese
Chocolate hamper in basket	minus 0.45	0.64	substitutes, not complements
Corporate buyer	0.60	1.82	offices build sets
Other items in basket (each)	0.20	1.22	bigger baskets get bigger

Reading it. A private buyer with only the wine hamper in the basket has odds of about 0.078 times 3.00, or 0.23, a probability of roughly 19 percent of adding cheese, against about 7 percent with an otherwise empty basket. A corporate buyer with the wine hamper is at about 30 percent. The chocolate line is what association rules would have missed: chocolate and cheese are substitutes in this shop, people pick one treat hamper, so recommending cheese to someone who has chosen chocolate is wasted screen space. And the rule "wine implies cheese" is partly a corporate effect in disguise, which is why the wine odds ratio of 3.0 is lower than the raw lift of about 4 you get from counting alone.

Fit one such model per product you might recommend, score the live basket at checkout, and show the two or three products with the highest predicted probability that are not already in the cart, with a floor so you never recommend anything below, say, 12 percent. The engineering side, including how to wire it into Solidus, is in building custom AI recommendations in Solidus; the statistical side is this paragraph. With hundreds of products the one model per product approach gets heavy and you move to matrix factorisation; for forty hampers it is exactly right.

Running it yourself

Data for the response model: one row per customer as of the campaign date, with the outcome (responded within the attribution window, usually four to six weeks, matched by code or email address) and predictors computed strictly from information available before the mailing. That last clause is where most first attempts leak. For the basket model you need one row per order with a flag per product.

A minimal fit in Python:

import numpy as np, pandas as pd
import statsmodels.api as sm

cols = ["months_since_last", "orders_24m", "gift_buyer",
        "corporate", "opened_last_spring", "aov_per_10"]
X = sm.add_constant(df[cols])
model = sm.Logit(df["responded"], X).fit()
print(model.summary())
odds_ratios = np.exp(model.params)

df["p_hat"] = model.predict(X)
df["decile"] = pd.qcut(df["p_hat"].rank(method="first"), 10,
                       labels=range(10, 0, -1)).astype(int)  # decile 1 = best

Timing: with clean order data, a first model in two days, a validated one with a decile table and a documented depth recommendation in about two weeks. The checks before you trust it: a holdout of 20 to 30 percent that never touches the fit; the holdout gains curve within a couple of points of the training curve; VIFs under 5; every sign explainable to a non statistician in one sentence; a calibration plot that hugs the diagonal; and a leakage hunt, where you list every variable with the exact date its value became known.

Pitfalls

Optimising response instead of profit. The top decile by response is not always the top decile by margin. Gift buyers respond well and buy the £45 hamper; corporate accounts respond less often and buy twelve. If margin varies a lot by segment, model expected value, or at least compute $m$ per decile.

Leakage from the future. Any predictor measured after the mailing date, even innocently (a "customer status" field updated when they order), produces a beautiful gains curve in the office and a useless one in the field. "Opened the spring email" is fine if it means last spring's; it is cheating if it means this spring's. If a model looks too good, it is.

Mailing the model into itself. Once you mail only the top four deciles, next year's data has responses only from those four, and a model fitted on it learns nothing about deciles 5 to 10. Always keep a small random holdout across the whole file, a few hundred people who get the brochure regardless of score. It costs a little and keeps the model honest year after year. The same logic applies to lookalike audiences on the ad platforms: a seed built only from people you already targeted teaches the platform to find more of the same.

Reading odds ratios as probability ratios. At a 3.9 percent base rate the difference is small; in the basket model at 19 to 30 percent it is not. An odds ratio of 3 is not "three times as likely".

Forgetting that association is not causation. The basket model tells you cheese and wine are bought together, not that showing cheese will sell more cheese. The recommendation widget is a hypothesis; an A/B test on the checkout, of the kind I describe in cart abandonment is a five figure problem, is the proof.

Too many variables. Forty predictors on 788 responders is overfitting waiting to happen. Ten is plenty. If the team insists on everything, use a penalised fit (ridge or lasso) and keep the holdout sacred.

How I do this for clients

The deliverable is a scored list, a decile table with a marked depth, and a one page memo that says how many people to contact, what it will cost, what it should return, and how wrong that estimate could be. Not a slide deck about machine learning.

It starts with the free workshop: ninety minutes on what customer and order data you actually have, which outcome we model (respond, buy, churn, add to basket), and what a good decision looks like. Then I spend two weeks of real work on my own account: feature table from your shop or CRM, fitted and validated model, decile table, gains chart, memo. If the model does not beat random by a margin worth acting on, I tell you so, and you have lost nothing.

If it does, and you want it running for every campaign rather than once, that becomes a data science engagement: a scoring job that runs nightly, a dashboard that answers one question ("how deep this time?"), and a random holdout that keeps it honest. For an e commerce shop the basket model goes straight into the store as a recommendation block, which is Solidus development I do myself. You own the code, the model and the data; nothing lives in a tool you have to rent from me.

Costs are on the pricing page in plain terms. Where the work is a campaign programme rather than a one off model, I also work on a commission basis tied to the incremental profit the holdout proves, which aligns my incentive with the column that matters.

Questions to ask before you approve the next mailing

• What is the breakeven response rate for this campaign, and which deciles clear it?
• Show me the gains chart on the holdout sample, not on the training data.
• Is there a random control group across the whole file, and how big is it?
• Which variables are in the model, and on which date was each one measured?
• Are we ranking by probability of response or by expected profit, and does it matter here?
• For the recommendations: which product pairs are substitutes, and are we suppressing them?

This series is inspired by Mike Grigsby's Marketing Analytics (Kogan Page). The explanations, examples and numbers here are my own.