Poisson regression: modelling how many times customers do things

The customer who ordered seven times

Every December the same thing happens at Northgate Hampers, an invented gift hamper shop in Cork run by Priya, at northgatehampers.example. Someone notices a customer who has ordered again and again, gets excited, and wants to build a VIP programme around people like her. This year it is a woman in Galway who has placed seven orders in two years. Her first hamper was a €75 gift for a colleague, she opens most of the emails, and she has never bought anything for herself. Is she remarkable? Is she the tip of a segment worth chasing? Or is she simply what happens when you have a few thousand customers and count for long enough?

That is a counting question, and counting questions have their own kind of model. Orders per customer per year, support tickets per account per month, stays per guest per season, quotes raised per workspace per week: none is a yes or no, and none is a smooth quantity like revenue. They are whole numbers, they start at zero, they cannot go below it, and most of the mass sits at the bottom with a long thin tail. Pushing them through ordinary least squares as if they were revenue is one of the most common quiet errors I see in marketing analytics.

This chapter is about the right tool: Poisson regression, its more forgiving cousin the negative binomial, and what to do when your zeros come from two different places. All the numbers are made up for the example and rounded so the reasoning is easy to follow.

Where this sits in the series

This is chapter 6 of 21 in my Marketing Analytics series, in the part on techniques where a single outcome is explained by a set of drivers. The previous chapter, polynomial distributed lags: how long does marketing keep working?, dealt with a continuous outcome spread over time. This one moves to counts. The next, logistic regression, lift charts and market basket analysis, takes the other common case, an outcome that is simply yes or no. Read the three together and you have most of the regression toolbox a small business will ever need.

Why ordinary regression stumbles on counts

Ordinary least squares assumes the outcome can wander freely in both directions around the line, with roughly the same spread everywhere. Counts break both assumptions, and a third one besides.

The floor. A straight line fitted to orders per year will happily predict minus 0.3 orders for a low value gift buyer with weak engagement. There is no such customer. You can clip the prediction at zero, but then your model is no longer the model you fitted.

The spread. For counts, the variance grows with the mean. Customers who order about once a year scatter between zero and three. Corporate accounts who order five times a year scatter between one and twelve. Least squares assumes constant variance, so it weighs the noisy heavy buyers the same as the quiet ones, and its standard errors come out wrong, usually too optimistic. You end up confident about effects that are not there.

The shape. Counts with a low mean are skewed. The average Northgate customer orders about 1.4 times a year, but the most common count is zero and a handful of accounts order fifteen times. A symmetric error term does not describe that.

I argued in counting and summing beats machine learning that most business questions are answered by counting well before they need anything clever. Poisson regression is what counting well looks like once you want to know why the counts differ.

The Poisson distribution and the log link

The Poisson distribution describes the number of events in a fixed window when events arrive independently at a steady average rate.¹ If a customer's underlying rate is $\lambda$ orders per year, the probability of seeing exactly $k$ orders is:

$$P(Y = k) = \frac{\lambda^{k} e^{-\lambda}}{k!}, \qquad k = 0, 1, 2, \ldots$$

Here $Y$ is the count, $\lambda$ (lambda) the expected number of events in the window, $k$ the count you actually observe, $e$ Euler's number, about 2.718, and $k!$ is k factorial, $k \times (k-1) \times \cdots \times 1$, with $0! = 1$. The distribution has one parameter, and that parameter is both its mean and its variance.¹ Remember that, because it is the property that bites later. Distributions, the Poisson among them, are explained from scratch in Foundations: distributions.

Worked example: one customer's chances

Take a customer with a rate of $\lambda = 1.4$ orders a year. The chance of no orders in a year:

$$P(Y = 0) = \frac{1.4^{0} e^{-1.4}}{0!} = e^{-1.4} = 0.247$$

Here $1.4^0 = 1$ and $0! = 1$, so only $e^{-1.4}$ is left, which is 0.247. The chance of exactly two orders:

$$P(Y = 2) = \frac{1.4^{2} e^{-1.4}}{2!} = \frac{1.96 \times 0.247}{2} = 0.242$$

Here $1.4^2 = 1.96$ and $2! = 2$. So a perfectly ordinary customer with that rate has a one in four chance of not ordering at all in a given year. A zero is not proof that someone has left.

Letting the rate depend on the customer

Poisson regression lets each customer have their own rate. The model does not use the rate directly, because a sum of drivers can be negative and a rate cannot. It uses the rate's logarithm:²

$$\log(\lambda_i) = \beta_0 + \beta_1 x_{1i} + \beta_2 x_{2i} + \cdots + \beta_p x_{pi}$$

Here $\lambda_i$ is the expected count for customer $i$, $\beta_0$ the intercept, $x_{1i}$ to $x_{pi}$ that customer's $p$ characteristics (first order value, whether the first purchase was a gift, whether they open emails, region) and $\beta_1$ to $\beta_p$ the coefficients we estimate. This is the log link. Whatever the right hand side adds up to, its exponential is positive, so the predicted rate is always a legitimate rate. The log and why it turns sums into products are in Foundations: logarithms, and regression itself in Foundations: regression.

The log link also changes how you read a coefficient. In ordinary regression it is an amount added. Here it is a multiplier. Exponentiate the coefficient and you get the incidence rate ratio:²

$$\text{IRR}_j = e^{\beta_j} = \frac{\lambda(x_j + 1)}{\lambda(x_j)}$$

Here $\text{IRR}_j$ is the incidence rate ratio for driver $j$, $\beta_j$ its coefficient, and $\lambda(x_j + 1)$ divided by $\lambda(x_j)$ the factor by which the expected rate changes when $x_j$ goes up by one unit, everything else held fixed. An IRR of 1.70 means 70% more orders per year. An IRR of 0.70 means 30% fewer. Multiplying percentages is set out in Foundations: percentages.

Multiplicative effects are how most marketing behaves. A good email programme does not add 0.4 orders to everyone. It lifts everyone's rate by a proportion, so the corporate account gains more orders in absolute terms than the occasional gift buyer. The model says so without being told.

Exposure: not everyone has been a customer for the same time

There is a trap in the data before you fit anything. A customer who joined 23 months ago has had 23 months to order. One who joined four months ago has had four. Count orders per customer and the old ones look like heavy buyers, and the model learns that tenure causes loyalty, which is backwards.

The fix is an exposure offset: model the rate per unit of time by including the log of each customer's observed time as a term whose coefficient is fixed at one.²

$$\log(\lambda_i) = \log(t_i) + \beta_0 + \beta_1 x_{1i} + \cdots + \beta_p x_{pi}$$

Here $t_i$ is the exposure, the number of years customer $i$ has been observable, and the other symbols are as before. Moving $\log(t_i)$ to the left gives $\log(\lambda_i / t_i)$, the log of the yearly rate, which is what we actually want to explain. The step uses $\log a - \log b = \log(a/b)$, from Foundations: logarithms.

Every count model I build for a client has an offset, and I check the offset before I check anything else. For a hotel group the exposure would be the number of seasons a guest has been in the database. For a SaaS product, the months a workspace has been live.

Worked example: two customers, one rate

Two Northgate customers each have three orders. One has been a customer for three years, the other for one. Without an offset they look identical. With it:

$$\frac{3}{3} = 1.0 \qquad \frac{3}{1} = 3.0$$

Here the numerators, $3$ in both, are the orders and the denominators, $3$ and $1$, the years observed, so the first customer orders once a year and the second three times. Same count, a threefold difference in rate.

The Northgate model: reading incidence rate ratios

Here is a fitted model on 4,180 Northgate customers, each observed for between six months and two years. The outcome is orders, the offset is the log of years observed, and the drivers are the ones Priya's team argues about at Christmas. The reference customer lives outside Connacht, buys for themselves, and had a €40 first order, no email opens in the first 90 days and no corporate account.

Driver	Coefficient	Standard error	IRR	95% interval	Plain reading
Intercept	minus 0.15	0.08	0.86	0.74 to 1.01	The reference customer orders 0.86 times a year
First order value, per €10 above €40	0.045	0.009	1.046	1.03 to 1.06	Every extra €10 in the first basket lifts the yearly rate by about 4.6%
First purchase was a gift	minus 0.36	0.06	0.70	0.62 to 0.79	Gift buyers reorder at 70% of the self buyer rate
Opened an email in the first 90 days	0.53	0.06	1.70	1.51 to 1.91	Early email engagement goes with a 70% higher rate
Connacht rather than the rest of Ireland	minus 0.15	0.07	0.86	0.75 to 0.99	Customers in Connacht order 14% less often, and the interval nearly touches 1
Corporate account	0.88	0.10	2.41	1.98 to 2.93	Corporate accounts order at roughly two and a half times the rate

Each interval is the coefficient plus and minus 1.96 standard errors, exponentiated. For the gift row, $-0.36 \pm 1.96 \times 0.06$ runs from $-0.478$ to $-0.242$, and $e^{-0.478} = 0.62$ and $e^{-0.242} = 0.79$. Confidence intervals are in Foundations: confidence intervals.

Read the table from the top. The intercept, exponentiated, is 0.86 orders a year: the anchor everything else multiplies. The first order value is per €10, so a €75 first order is 3.5 steps above the €40 reference and contributes $1.046^{3.5} \approx 1.17$, a 17% higher rate. The gift IRR is below one: gift buyers come back less, which every hamper seller suspects and few have quantified. Email engagement is the largest effect the shop can influence, at 1.70. The Connacht effect is real but small and its interval runs up to 0.99, so I would not build a regional strategy on it. The corporate effect is large and expected.

Worked example: the Galway customer

She is a gift buyer in Galway, so in Connacht, with a €75 first order who opens emails and has no corporate account. Multiply it through:

$$\lambda = 0.86 \times 1.046^{3.5} \times 0.70 \times 1.70 \times 0.86 \approx 1.03 \text{ orders per year}$$

Here $0.86$ is the intercept rate, $1.046^{3.5}$ the first order value effect for 3.5 steps of €10, $0.70$ the gift effect, $1.70$ the email effect and the second $0.86$ the Connacht effect. Over the two years she has been observed, the model expects about 2.06 orders. She placed seven, roughly 3.4 times the expected number. Whether that is remarkable depends entirely on the next section.

Does the Poisson fit? The overdispersion check

The Poisson insists that the variance equals the mean. Real customers rarely oblige. Some are quietly loyal, some bought once by accident, some are small businesses hiding behind a personal email address. That hidden variety makes the counts spread wider than a Poisson with the same mean allows. It is called overdispersion, and in customer data it is the rule rather than the exception.²

The first check is a picture: the observed counts of orders in a year against what a Poisson with the same mean of 1.4 predicts.

The horizontal axis is the number of orders a customer placed in a year, the vertical axis the number of customers with that count. The Poisson line is $4{,}180 \times P(Y = k)$ with $\lambda = 1.4$. The pattern is the classic signature: more zeros than expected, fewer customers in the middle and a fat right tail. The Poisson predicts two customers with seven orders and almost none with eight or more. The shop has 80 customers with seven or more.

The second check is a number. After fitting, compute the Pearson dispersion statistic:²

$$\hat{\phi} = \frac{1}{n - p} \sum_{i=1}^{n} \frac{(y_i - \hat{\lambda}_i)^2}{\hat{\lambda}_i}$$

Here $\hat{\phi}$ (phi hat) is the dispersion estimate, $y_i$ the observed count for customer $i$, $\hat{\lambda}_i$ the model's predicted count, $n$ the number of customers, $p$ the number of estimated coefficients, and $\sum$ adds over all customers. Each term asks how far a customer sits from the prediction, in units of the variance the Poisson allows. If the Poisson is right, $\hat{\phi}$ is close to 1. Variance and the reason for dividing by $n - p$ rather than $n$ are in Foundations: spread.

Worked example: the dispersion of the raw counts

With no drivers in the model, every prediction is the overall mean, and the statistic is simply the variance divided by the mean. For the counts in the chart, counting 8 or more as eight:

$$\hat{\phi} = \frac{2.75}{1.39} = 1.98$$

Here $2.75$ is the variance of the 4,180 counts and $1.39$ their mean. The variance is about twice the mean, and a little more than that in reality, because some of the 45 customers in the top group ordered more than eight times. After the drivers go in, which soak up some of the spread (the corporate flag alone explains a lot of the tail), the example's fitted model still has $\hat{\phi} = 1.9$. My working rule: anything above about 1.3 and I stop trusting the Poisson standard errors, above 1.5 I switch model.

What goes wrong if you do not? The coefficients are usually still roughly right. The standard errors are not. They are too small by a factor of about $\sqrt{\hat{\phi}}$, which is what a quasi Poisson correction scales them by.² With $\hat{\phi} = 1.9$, $\sqrt{1.9} = 1.38$, so every interval in the table should be about 38% wider. The Connacht effect's upper limit moves from 0.99 to about 1.04, and the effect is no longer significant. That is exactly the kind of effect people build campaigns around and then wonder why the campaign did nothing.

The negative binomial: how surprised to be

The negative binomial model is the standard remedy. It keeps the log link and the IRR reading, so nothing above is wasted, but it lets the variance grow faster than the mean:³

$$\operatorname{Var}(Y_i) = \lambda_i + \alpha \lambda_i^{2}$$

Here $\operatorname{Var}(Y_i)$ is the variance of customer $i$'s count, $\lambda_i$ its mean as before, and $\alpha$ (alpha) a dispersion parameter estimated from the data. When $\alpha = 0$ you are back at the Poisson. Variance itself is in Foundations: spread. For Northgate the example's fitted $\alpha$ is 0.48 once the drivers are in. Refit with the negative binomial and the IRRs barely move, but the intervals widen honestly and the tail probabilities change dramatically.

Worked example: seven orders under each model

Back to Galway. Her expected count over two years is 2.06. Under the Poisson, the probability of seven or more orders is $P(Y \ge 7) = 0.005$, about one customer in 190. She looks like a star.

Under the negative binomial with $\alpha = 0.48$, her variance is:

$$2.06 + 0.48 \times 2.06^{2} = 2.06 + 2.04 = 4.10$$

Here $2.06$ is her expected count and $0.48$ the dispersion parameter. Nearly double the Poisson's 2.06, and the tail is fatter to match: the probability of seven or more becomes 0.037, about one in 27. If the shop had 4,180 customers just like her, about 155 would reach seven orders by ordinary variation in loyalty alone. She is a good customer and deserves a thank you, but she is not a segment, and a VIP programme designed around her would be designed around noise. That is the practical difference between the two models: the negative binomial tells you how surprised to be.

Zero inflation: when the zeros come from two places

Zero inflation is a different problem from overdispersion. Overdispersion is about spread. Zero inflation is about zeros coming from two populations.⁴

At a hamper shop, some zeros are customers who may well order again and simply have not yet. They sit on the Poisson or negative binomial curve. Other zeros are people who bought one gift for a leaving colleague, will never think of the shop again and could not be moved by any email. They are structural zeros. A zero inflated model fits two parts: a logistic part for the probability of being a structural never again customer, and a count part for everyone else.⁴ It is worth it when the share of zeros is far above what even the negative binomial predicts and you have a plausible driver for the never again group, such as a corporate gift shipped to a third party address.

Here is how I decide:

Read it from the top: each diamond is a check, and the box after it is what to do with the answer. Most of the work I do ends at the negative binomial.

An example of each ending. In the Northgate example the negative binomial predicts about 1,610 zeros against 1,560 observed. That is close enough. The zeros are explained by variety among customers, not by a hidden never again population, so the decision stops there. Now picture a hotel group whose guests often book once through a booking platform and never leave an email address. For direct bookings, those guests are almost all structural zeros, and a zero inflated model with "booked through a platform" as its driver would be clearly better.

From model to targeting list

A fitted count model is only useful if it changes who you talk to. Three lists come out of it almost for free.

The expected rate list. Every customer gets a predicted $\hat{\lambda}_i$ per year. Multiply by the average contribution per order and you have expected yearly contribution, the honest basis for how much to spend keeping each person. This connects to the retention maths: a small lift in the rate for the middle of the list is worth more than a large lift for a handful at the top.

The surprise list. Compare what each customer did with what the model expected. The Pearson residual $(y_i - \hat{\mu}_i)/\sqrt{\hat{\mu}_i + \alpha \hat{\mu}_i^2}$ standardises the gap, where $y_i$ is the observed count, $\hat{\mu}_i$ the expected count over the customer's own exposure and $\alpha$ the dispersion parameter. Customers far above expectation are worth a human look, not a campaign: a good share turn out to be small companies ordering under a personal address, which means a phone call and a corporate account rather than a discount code.

The lever list. Because email engagement carries an IRR of 1.70, the customers where an engagement nudge is worth most are those with a high baseline rate and no engagement yet. The expected gain for each is $\hat{\lambda}_i \times (1.70 - 1)$ extra orders a year if you could move them, which you cannot fully, so I discount it.

Worked example: five customers on the list

Five customers, each observed for two years, scored with the example model and $\alpha = 0.48$:

Customer	Profile	Expected per year	Expected over two years	Observed orders	Residual	Action
A	Cork corporate account, €120 first order, engaged	5.05	10.1	11	0.1	Keep. Account manager already in place
B	Galway gift buyer, €75, engaged	1.03	2.06	7	2.4	Thank you note. Check for a hidden business
C	Kilkenny self buyer, €95, not engaged	1.10	2.20	2	minus 0.1	Engagement nudge: highest expected gain
D	Waterford gift buyer, €35, not engaged	0.59	1.17	0	minus 0.9	Leave alone. Likely a structural zero
E	Sligo self buyer, €60, engaged	1.38	2.75	0	minus 1.1	Win back email. Expected but silent

For customer B, the residual is $(7 - 2.06)/\sqrt{4.10} = 4.94/2.02 = 2.4$, using the variance worked out above. Notable, not extraordinary, which is what the tail probability already said. Customer C is the one most marketers would ignore and the model would prioritise: a decent baseline with the biggest lever untouched, worth $1.10 \times 0.70 = 0.77$ extra orders a year at full effect. Customer D would get a discount from a naive campaign, and it would be wasted.

Running it yourself

You need one row per customer with the count, the exposure in years and the drivers as they were known at the start of the window. That last point matters: use the first order value and the first 90 days of email behaviour, not lifetime figures, or you leak the outcome into the inputs. In SQL it is one aggregation:

select c.id,
       count(o.id) as orders,
       extract(epoch from now() - c.first_order_at) / 31557600.0 as years_observed,
       (c.first_order_total - 40) / 10.0 as first_value_10,
       c.first_order_is_gift::int as gift,
       c.opened_email_first_90d::int as email_open,
       (c.province = 'Connacht')::int as connacht,
       c.is_corporate::int as corporate
from customers c
left join orders o on o.customer_id = c.id and o.completed_at > c.first_order_at
where c.first_order_at < now() - interval '6 months'
group by c.id;

Fitting takes seconds. In Python with statsmodels, the offset goes straight into the generalised linear model, and its negative binomial family uses the variance $\mu + \alpha\mu^2$ from above:³⁵

import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf

formula = "orders ~ first_value_10 + gift + email_open + connacht + corporate"
pois = smf.glm(formula, data=df, family=sm.families.Poisson(),
               offset=np.log(df["years_observed"])).fit()
phi = pois.pearson_chi2 / pois.df_resid
print(f"dispersion {phi:.2f}")          # near 1.0 is fine, above 1.5 switch

nb = smf.glm(formula, data=df, family=sm.families.NegativeBinomial(alpha=0.48),
             offset=np.log(df["years_observed"])).fit()
print(np.exp(nb.params))                 # incidence rate ratios

In R the same is glm(..., family = poisson, offset = log(years_observed)) followed by MASS::glm.nb.⁶ Estimating alpha from the data instead of fixing it takes one more line in either language.

Before you trust it, run four checks. Plot observed against predicted counts by decile of predicted rate: the points should sit on the diagonal. Compare the predicted share of zeros with the observed share. Hold out the most recent six months and see whether the model's ranking of customers holds on orders it never saw. And look at the largest residuals by hand: if the top ten are all one kind of customer, you are missing a driver, not looking at noise.

Time budget: a day for the data, half a day for the modelling, a day to turn it into lists people will actually use. The modelling is the smallest part, which is true of nearly everything in this series.

Pitfalls

Counting the wrong window. If the outcome window overlaps the period the drivers were measured in, the model explains the past with the past. Freeze the drivers at the start and count from there.

Forgetting the offset. I have seen a model that proudly showed tenure as the strongest driver of orders. Of course it was: those customers had been counted for longer. The offset removes this in one line and the whole story changes.

Reading IRRs as additive. An IRR of 1.70 on a baseline of 0.3 orders is $0.3 \times 0.70 = 0.21$ extra orders a year. The same IRR on a baseline of five is $5 \times 0.70 = 3.5$. Always multiply through to the customer's own baseline before promising anyone a result.

Trusting Poisson intervals when the dispersion is 2. The coefficients look fine, the significance stars look fine, and some of the significant effects are not. Check the dispersion every time. It is one number.

Chasing the tail. Heavy buyers exist in every negative binomial, by construction. Before you announce a segment, ask how many customers at that level the model already expects. Usually it expects about as many as you found.

Adding zero inflation because it fits better. A zero inflated model almost always fits a little better, because it has more parameters. Use it when you can name the never again population and treat it differently. Otherwise it is complexity with no decision attached.

How I do this for clients

The deliverable is a scored customer list and a one page decision memo, not a model file. It starts with the free workshop, where we agree what to count (orders, stays, tickets, quotes), what the exposure is, and the four or five drivers that were actually known at the start of each customer's window. Then the first two weeks of work are on me, before you commit to anything.

The data I need is unglamorous: an export of customers with their first order details and a table of later events, or read access to the shop or CRM database. For a Solidus shop it is one query against the orders and users tables. For a Rails SaaS it is workspaces and their usage events. I build the exposure properly, fit the Poisson, run the dispersion check, move to the negative binomial when the data demands it, and only go zero inflated when we can name the group.

What you get: every customer scored with an expected yearly rate and an error band, the three lists above as CSV or straight into your email tool, the IRR table in plain language, and a memo that says what the model recommends, what it would be worth if the lever works half as well as the coefficient suggests, and where it is uncertain. You own all of it, including the code.

After the two weeks, the work runs on the monthly terms on the pricing page as Data Science. For an eCommerce store there is also the commission model, a monthly base rate or a commission on measured growth, whichever is higher, which suits the follow on Performance Marketing if you prefer paying for outcomes. I am one accountable person from data to campaign, which is the point.

Questions to put to your own team

• What exactly is the count, and over what window per customer? If nobody can name the exposure, the model has no offset.
• Were the drivers measured before the counting window started, or during it?
• What is the dispersion statistic of the fitted Poisson model, and what did you do when it came out above 1.5?
• Are the effects reported as multipliers, and have they been multiplied through to a real customer's baseline?
• How many customers with this many orders does the model already expect? Is the "segment" larger than that?
• Does the predicted share of zeros match the observed share, and if not, can you name who the extra zeros are?
• Has the customer ranking been checked on a period the model never saw?
• Which list changed because of this work, and who is working through it?

The series is inspired by Mike Grigsby's Marketing Analytics.⁷ The explanations, examples and numbers here are my own.

Footnotes

1. NIST/SEMATECH, e-Handbook of Statistical Methods, section 1.3.6.6.19, Poisson Distribution, https://www.itl.nist.gov/div898/handbook/eda/section3/eda366j.htm

2. Penn State Eberly College of Science, STAT 504 Analysis of Discrete Data, Lesson 9: Poisson Regression (the log link, offsets, overdispersion and the scale parameter), https://online.stat.psu.edu/stat504/Lesson09

3. statsmodels developers, statsmodels.genmod.families.family.NegativeBinomial, statsmodels documentation, https://www.statsmodels.org/stable/generated/statsmodels.genmod.families.family.NegativeBinomial.html

4. UCLA Office of Advanced Research Computing, Zero-Inflated Poisson Regression, R Data Analysis Examples, https://stats.oarc.ucla.edu/r/dae/zip/

5. statsmodels developers, Generalized Linear Models, statsmodels documentation, https://www.statsmodels.org/stable/glm.html

6. UCLA Office of Advanced Research Computing, Negative Binomial Regression, R Data Analysis Examples, https://stats.oarc.ucla.edu/r/dae/negative-binomial-regression/

7. Mike Grigsby, Marketing Analytics: A Practical Guide to Improving Consumer Insights Using Data Techniques, Kogan Page, 2022, ISBN 9781398608191, https://www.netgalley.co.uk/catalog/book/267990