Tools of segmentation: k means, latent class analysis and going beyond RFM

The two customers RFM said were the same

Last November two orders landed at The Gift Bow within an hour of each other. The first came from an office manager in Leeds: twelve of the same hamper, twelve delivery addresses, a company name in the billing field, a request for a VAT invoice and a card with the firm's logo. £540. The second came from a woman in Bath who bought three hampers for her three sisters, one gift message each, £190. Both were first orders. Both were recent. Both sat in the top fifth of order values, because most orders in the shop are a single hamper between £40 and £60.

The RFM scoring I had set up as a quick win put them in the same cell: recency 5, frequency 1, monetary 5. The automation attached to that cell sent both of them a "welcome, here is ten percent off your next hamper" email in the second week of December. The office manager did not need ten percent off. She needed someone to ring her in October next year and ask how many she wants this time. The woman in Bath did not need a discount either. She needed a reminder in late February that Mother's Day is coming. RFM saw two numbers and one behaviour. There were two behaviours, and frankly two businesses, inside one shop. This chapter is about the tools that tell them apart. The shop is real, it is my own Solidus demo at hampers.keferboeck.com; the numbers are illustrative.

Where this sits in the series

This is the last chapter of part three, interrelationship techniques. The part opened with simultaneous equations, moved through principal components and factor analysis, and then set out the thinking in Segmentation: strategy before algorithms, which argued that you decide what a segment is for before you touch an algorithm. This chapter picks up the algorithms. Next we open part four, media and loyalty, with Modelling the value of marketing communications.

What a good segmentation has to deliver

Four things, and a segmentation that fails any one of them fails.

Separation: segments should be far apart on the things that drive behaviour. If two segments differ only slightly, the marketing for each is the same, and what you really have is one segment with a line drawn through it.

Cohesion: customers inside a segment should be alike enough that one message, one offer and one landing page work for most of them.

Interpretability: the person who runs the email programme should be able to describe every segment in a sentence without a table. If the sentence needs the words "component two", start again.

Actionability: each segment needs at least one thing you would do differently for it. A different channel, offer, timing or product. If you cannot name it, the segment is a statistical curiosity.

The first two are measurable. The last two are judgement, and they are where most segmentations quietly die: beautifully separated clusters that nobody in the business can use.

The toolbox

Hierarchical clustering builds a tree. The agglomerative version starts with every customer as their own cluster and merges the closest pair, over and over, until one cluster remains; you then cut the tree at the height that gives you the number of groups you want. Ward's linkage, which merges the pair that increases the total within cluster variance the least, produces compact groups and is my default. The strength is the dendrogram, which shows the whole structure at every possible number of groups. The weakness is cost: it needs the distance between every pair of customers, fine for 6,000 and unworkable for 6 million.

k means picks k centre points, assigns every customer to the nearest centre, moves each centre to the mean of its members, and repeats until nothing changes. It is fast, it scales, it is in every library, and it is what most people mean by "clustering". It also has firm opinions: it wants numeric variables on comparable scales, it makes roughly spherical groups of similar spread, and it will find k clusters whether or not the data contains any.

Latent class analysis takes a different view. It assumes the population is a mixture of a few unobserved classes, that within each class the observed variables are independent, and that each class has its own probability of each response. Membership is probabilistic: a customer is 84 percent class B and 16 percent class A. It handles categorical variables natively, which k means cannot, and it gives you a principled way to choose the number of classes.

Mixture models generalise this to continuous variables: a Gaussian mixture is the probabilistic cousin of k means, with ellipses instead of spheres and soft membership instead of hard.

Decision trees are not a segmentation method here; they are the profiling tool. A shallow tree that predicts membership from a handful of plain variables gives you the shortest description of each group and rules you can type into an email tool.

The maths, briefly

k means minimises the sum of squared distances between every customer and the centre of the cluster they belong to:

$$J = \sum_{j=1}^{k} \sum_{x_i \in C_j} \lVert x_i - \mu_j \rVert^2$$

Here $k$ is the number of clusters, $C_j$ the set of customers assigned to cluster $j$, $x_i$ the vector of variables for customer $i$, $\mu_j$ the centre (the mean) of cluster $j$, and the double bars denote Euclidean distance. $J$ is the within cluster sum of squares, WCSS for short, the number on the elbow chart below. The algorithm only guarantees a local minimum, which is why you run it from many random starts and keep the best.

The classic RFM score turns three quintile ranks into one three digit number:

$$\text{RFM}_i = 100\,R_i + 10\,F_i + M_i$$

$R_i$ is the recency quintile of customer $i$, with 5 for the most recent fifth, $F_i$ the frequency quintile and $M_i$ the monetary quintile. A 555 is your best cell; a 111 has probably left. It is a ranking of value, not a description of behaviour, and the two customers in the opening scene both scored 515.

Latent class analysis maximises a likelihood. In words: the probability of one customer's pattern of answers is the sum, over all classes, of the probability of being in that class times the probability of that pattern given the class; within a class the variables are assumed independent, so that conditional probability is a plain product:

$$L(\theta) = \prod_{i=1}^{n} \sum_{c=1}^{k} \pi_c \prod_{v=1}^{V} P(x_{iv} \mid c)$$

$n$ is the number of customers, $k$ the number of classes, $\pi_c$ the share of the population in class $c$, $V$ the number of observed variables, and $P(x_{iv} \mid c)$ the probability that a member of class $c$ shows the value customer $i$ showed on variable $v$. After fitting, Bayes' rule gives every customer a posterior probability of belonging to each class:

$$P(c \mid x_i) = \frac{\pi_c \prod_{v} P(x_{iv} \mid c)}{\sum_{c'} \pi_{c'} \prod_{v} P(x_{iv} \mid c')}$$

That posterior is the practical difference from k means. k means says "you are in cluster 2". LCA says "you are 0.84 cluster 2 and 0.16 cluster 1", and the 0.16 is information you can act on.

For choosing k I lean on two measures. The elbow uses $J$ above. The silhouette compares, per customer, the mean distance to their own cluster with the mean distance to the nearest other cluster:

$$s(i) = \frac{b(i) - a(i)}{\max\{a(i),\, b(i)\}}$$

$a(i)$ is the mean distance from customer $i$ to the other members of their own cluster, $b(i)$ the mean distance to the members of the closest other cluster. Near 1 means well placed, near 0 means on a boundary, below 0 means probably misassigned. The average over all customers gives one number per k.

The worked example: four segments at The Gift Bow

In Principal components and factor analysis for marketers I reduced fourteen behavioural variables, built from 24 months of orders by 6,200 customers, to four components. Occasion gifting: gift messages, recipient address different from billing, orders clustering around Christmas, Mother's Day and birthdays. Corporate scale: units per order, several delivery addresses on one order, company field filled, invoice requested, weekday ordering. Self indulgence: shipped to own address, cheese and wine, short gaps between orders, interest in the subscription box. Price sensitivity: discount codes, sale period purchases, price paid relative to the range median, checkout abandonment at the shipping step.

Component scores are already standardised, which removes the first k means trap. I ran k means for k from 2 to 8, fifty random starts each.

Elbow chart: the drop from k=3 to k=4 is still large; from k=4 onwards the curve flattens.

The elbow is visible but, as always, not sharp. The silhouette and a look at cluster sizes settle it:

k	WCSS	Mean silhouette	Smallest cluster
2	17,900	0.34	2,310
3	13,600	0.31	1,180
4	10,900	0.38	430
5	9,700	0.33	190
6	8,900	0.29	110

Read it row by row. Two clusters give a decent silhouette, but only because they split "gifting" from "everything else", which we knew. Three lose silhouette because the corporate buyers get pulled into the gift cluster and stretch it. Four gives the best silhouette, and the smallest cluster is 430 customers, seven percent, small but real. Five carves 190 customers out of the self buyers on price sensitivity alone, and six produces groups too small to build a campaign around. Four it is, and four matched what the strategy chapter said the business could act on.

Here is the profile.

Segment	Customers	Share	Orders per year	Avg order £	Units per order	Gift message	Other address	Discount used	Peak months
A Occasion gifters	2,480	40%	1.6	62	1.2	91%	94%	22%	Dec, Mar
B Corporate senders	430	7%	1.3	1,140	22	88%	100%	9%	Nov, Dec
C Treat yourself regulars	1,240	20%	5.8	48	1.1	6%	8%	31%	flat
D Bargain occasionals	2,050	33%	1.1	34	1.0	55%	60%	78%	Jan, Nov

And the centroids, with each segment's mean component score expressed as a percentile across all customers, so 50 is average and 90 is near the top:

Segment	Occasion gifting	Corporate scale	Self indulgence	Price sensitivity
A Occasion gifters	82	38	21	41
B Corporate senders	70	97	15	30
C Treat yourself regulars	12	27	91	52
D Bargain occasionals	58	22	30	88

Segment A is the shop's centre of gravity: two fifths of customers, a gift message on nearly every order, one and a half occasions a year, discount use below average. They buy on a date, not on a price.

Segment B is seven percent of customers and, when I multiply the rows out, about half the revenue: 430 customers times 1.3 orders times £1,140 is roughly £637,000 against a total of about £1.3 million. RFM scored almost all of them frequency 1 or 2, because they order once a year, so on a frequency ranking they look like casual buyers. They are the most valuable relationships the shop has.

Segment C never sends a gift. They order cheese and wine for themselves every couple of months, read the product emails, and are the natural audience for the subscription box. Their discount use is a shade above average, but the peak month column says "flat": they are not waiting for the sale.

Segment D is the group the discount automation was built for: a third of the customer base and six percent of revenue, a gift message on about half of orders, one order a year, nearly four in five orders with a code, peaks in the January sale and around Black Friday. Some are gift buyers who found the shop through a voucher site. The name is a hypothesis, and I come back to that in the pitfalls.

Back to the two customers from the opening. The office manager in Leeds sits deep inside B, at percentile 98 on corporate scale. The woman in Bath is a textbook A. RFM saw 515 twice. The segmentation sees a corporate account to phone in October and an occasion gifter to remind in February. Neither should have been sent ten percent off.

Revenue share by segment: the smallest segment by headcount carries half the turnover.

The LCA comparison

To check that k means was not simply drawing tidy circles on noise, I refitted the same customers with latent class analysis on categorical versions of the raw variables: gift message, delivery address differs, company field filled, discount code used, ordered in November or December, ordered in more than one quarter, each yes or no, plus units per order in three bands. Seven indicators, all binary or three level, which is LCA's home turf.

Classes	Negative log likelihood	BIC	Smallest class	Entropy
2	20,480	41,091	2,190	0.81
3	19,870	39,941	1,040	0.79
4	19,560	39,391	410	0.83
5	19,535	39,410	160	0.74
6	19,520	39,450	90	0.70

The BIC bottoms out at four classes, and the entropy, a measure of how confidently customers are assigned (1 is perfectly certain), peaks there too. Five classes buys almost no likelihood and pays with a class of 160 customers and fuzzier membership.

Then the crosstab. The four classes line up with the four k means clusters: 81 percent of customers land in the matching class, and most of the 19 percent that move sit between A and D, occasion gifters who used a code once or bargain occasionals who wrote a gift message. The posteriors agree: 68 percent of customers have a maximum posterior above 0.9, 20 percent between 0.7 and 0.9, 12 percent below 0.7, almost all of the last group between A and D. That boundary is real. k means just hides it by drawing a hard line through it.

Three tools, three questions: RFM ranks value, k means partitions behaviour, LCA estimates the probability of belonging.

My rule: k means when the inputs are continuous and already cleaned into components; LCA when they are mostly yes or no answers, survey items or categories; a Gaussian mixture when you want soft membership on continuous data. Whatever you pick, run one of the others as a check. Agreement above 75 to 80 percent between two methods is the strongest evidence I know that the segments live in the data rather than in the algorithm.

What RFM cannot see and behavioural segments can

RFM is a good ranking of value at a point in time, the right tool for "who do we contact first" when the budget stretches to one mailing. It fails as a segmentation for three reasons that sit in its definition, not its implementation.

It has no needs in it. Recency, frequency and monetary value say what happened, not why. Two customers with identical scores may have bought for a wedding, a bereavement, a contract renewal or a Tuesday.

It has no cause of purchase. A gift buyer buys because a date is coming, a corporate buyer because a budget year is closing, a self buyer because the fridge is empty. Three triggers, three calendars, and a frequency quintile flattens all of them.

It has no response in it. What did each group do the last time you sent a discount, a product story, a reminder? With a segment field on the customer record I can look that up. In the demo, segment D's click rate on discount emails is three times segment A's, and A's click rate on the "Mother's Day is in three weeks" email is the highest in the file. Segment B does not open marketing emails at all; the October phone call has a reorder rate around 60 percent. Illustrative, but every one of those numbers is measurable once the segment exists.

Behavioural segmentation, built on what people did across categories, occasions, channels and offers, gives you the trigger, the need and the response, which is what you need to write different marketing for different people, the whole point of Segmentation is not demographics.

Running it in practice

Data. A customer table with 12 to 24 months of orders; line items with categories; whether delivery and billing addresses differ (not the addresses themselves); the gift message flag; discount code usage; the company field; email engagement if you have it. In a Solidus shop all of this sits in spree_orders, spree_line_items, spree_addresses and spree_adjustments. A few hundred customers is too few for k means to mean anything; from about a thousand you can start, from five thousand you can trust it.

Steps. Build one row per customer with 10 to 20 behavioural variables. Standardise, or reduce to components as in chapter 12. Run k means for k from 2 to 8 with many starts; plot WCSS and silhouette. Fit LCA or a Gaussian mixture as a second opinion. Pick k with the business in the room. Profile with a table and a shallow tree. Name the segments. Write a scoring script. Load the segment field into the shop and the email tool.

The scoring script is short, and it must reuse the fitted objects rather than refit:

import joblib, pandas as pd

scaler = joblib.load("scaler.pkl")   # fitted on the training window
pca = joblib.load("pca.pkl")         # the four components from chapter 12
km = joblib.load("kmeans_k4.pkl")    # centroids frozen at fit time
names = {0: "A", 1: "B", 2: "C", 3: "D"}

X = pd.read_sql("select * from customer_features", conn).set_index("customer_id")
Z = pca.transform(scaler.transform(X[pca.feature_names_in_]))
X["segment"] = [names[c] for c in km.predict(Z)]
X["distance"] = km.transform(Z).min(axis=1)   # far from every centre: flag for review
X[["segment", "distance"]].to_sql("customer_segments", conn, if_exists="replace")

Rescore monthly. Do not refit monthly: refitting moves the centroids and relabels everyone, which breaks every campaign report that references a segment. Refit once or twice a year and treat that as a small project with a migration table.

Checks before you trust it. Run k means from twenty seeds and compare the assignments with the adjusted Rand index; below 0.9 the solution is unstable and you should not ship it. Bootstrap: resample customers with replacement, refit, compare with the original; I want a median adjusted Rand index above 0.7. Split half: fit on a random half, score the other, compare profiles. Holdout in time: fit on months 1 to 18, score months 19 to 24, check that the profile table still reads the same. Predictive validity: does membership predict something the model never saw, such as response to last spring's campaign or next quarter's revenue? Segments that predict nothing outside their own inputs are a description, not a tool.

For a shop of this size, two to three weeks, and half of that on the feature table.

Pitfalls

Unscaled inputs. Order value in pounds next to a gift message flag of 0 or 1: k means will cluster on pounds and nothing else. Standardise, or better, cluster on components.

Spheres where the data has none. k means partitions space into cells around the centres. If the true structure is a long thin group, say regular self buyers spread along a spend gradient, it will chop that group in two. Mixture models soften this; a per cluster silhouette usually reveals it.

Rediscovering RFM. If the clustering variables are dominated by spend and frequency, the clusters will be "high, medium, low value" and you have built RFM with more steps. Keep the value variables out of the clustering and use them to profile afterwards. That is why segment B's revenue share is a finding and not an input.

Too many segments. Every segment needs a creative brief, a landing page variant, an email flow and an owner. Four is manageable for a small team. Eight is a spreadsheet nobody opens.

Static membership. People move. A segment D customer who buys three times at full price is now a C or an A. Score monthly and build a migration table: how many moved from where to where. A healthy programme moves people from D to A and from A to B.

Names become facts. "Bargain occasionals" is a hypothesis about 2,050 people generated from four numbers. Some are students. Some are gift buyers who found the shop on a voucher site and would happily pay full price for a hamper they trust. Test the name: send half of D a full price product story instead of a code and measure.

LCA has two of its own. Label switching: class 1 in one run is class 3 in the next, so match classes on their profiles, never on their index. Local independence: two near duplicate indicators, like "gift message" and "delivery address differs", inflate the number of classes; drop or merge one.

How I do this for clients

The data I need is read access to the order database or an export of it, 12 to 24 months, plus email engagement if it exists. Nothing beyond what the shop already holds, and never the addresses themselves.

The first month starts with the free workshop, half a day, in which we settle what the segments are for and which two or three actions the business could realistically take for each. Then two weeks of real work on me: the feature table, the components, k means and LCA side by side, the stability checks, the profile, the names. If you decide the result is not worth paying for, you keep the workshop notes and owe me nothing.

What you get: a scored customer table with a segment and a confidence for every customer, loaded back into your shop and your email tool; a one page profile per segment written for the people who will use it; a decision memo with the number that matters (in the demo, seven percent of customers are half the revenue and are being treated as one off buyers) and the error margin and risk around it; the scoring script, which you own; and a dashboard that answers one question, "how many customers are in each segment this month and where did they move from". After that, if you want it, I run the activation on commission: the field goes into the shop, the email tool and the ad platforms as one customer list per segment, with the October call list for B, the February reminder for A, the subscription push for C, and lookalike audiences seeded from A and B alone, because seeded from the whole file they would mostly find more of D.

The model build is a fixed piece of work; the terms are on the pricing page. The method sits in data science, the activation in performance marketing, and both start with the free workshop.

Questions to ask before you sign off a segmentation

• Which variables were used to cluster, and were spend and frequency among them?
• How was the number of segments chosen, and what did the silhouette and a second method say?
• Can each segment be described in one sentence without looking at a table?
• What will we do differently for each segment, and who owns that?
• How stable are the segments under resampling and across time?
• What is the plan for rescoring, and how will we see migration between segments?
• Which of the segment names are hypotheses we still have to test?

This series is inspired by Mike Grigsby's Marketing Analytics (Kogan Page). The explanations, examples and numbers here are my own.