Principal components and factor analysis for marketers

Twenty four columns of averages and no answer

Every autumn Seeblick Hotels, a small group of five houses around the lakes of the Salzkammergut, sends its guests a survey. Twenty four questions, each rated from 1 to 10: how warm the reception was, how quickly staff solved problems, whether the room was clean, how the breakfast tasted, whether the price felt fair. Last season 1,340 guests answered. Anna, who runs the group with her brother, sent me the spreadsheet with a one line email: "Everything is between 7.6 and 8.7. What do we fix first, and will it move direct bookings?"

That is the honest problem with survey data and with most customer databases. You do not have 24 pieces of information. You have perhaps four things, each measured six times with slightly different wording, plus noise. A guest who gives the reception a 9 gives "staff solved problems quickly" a 9 as well, because both questions ask about the same underlying thing: how the people treated her. Until you find those underlying things, every chart of 24 bars is the same four bars drawn over and over, and what to fix stays a matter of taste.

Principal component analysis and factor analysis are the two tools for this job. They take dozens of correlated variables and boil them down to a handful of dimensions you can name, score and act on. They look alike, they are often confused, and they answer different questions. This chapter covers both, using Anna's survey and, as a second example, the behavioural data of my hamper shop, which prepares the ground for chapter 14. The numbers throughout are illustrative.

Where this sits in the series

This is chapter 12 of 21 and the second chapter of part three, the interrelationship techniques (chapters 11 to 14). The previous chapter, Simultaneous equations: when marketing causes sales and sales cause marketing, closed part two by showing what happens when two variables cause each other. From here on there is no Y at all. We stop asking what predicts sales and start asking how variables hang together. The next chapter, Segmentation: strategy before algorithms, asks what a segment is for before anyone opens a clustering library, and chapter 14 then uses the components built here as raw material.

Two techniques that look alike and are not

Both methods start from the same object, the correlation matrix of your variables, a square table that says how much each pair moves together, and both end with a few new columns that replace the many old ones. The philosophies differ, and the difference matters when you interpret the result.

Principal component analysis (PCA) is geometry. It takes the cloud of data points in $p$ dimensions and finds the direction in which the cloud is widest. That is the first component. It then finds the widest direction at right angles to the first, the second component, and so on until it has $p$ new axes that describe the same cloud, just rotated. Nothing is assumed about why the variables correlate. There is no error term. PCA is a lossless rewriting of the data in which the interesting variance is pushed to the front so that you can drop the tail.

Factor analysis (FA) is a model. It assumes that a small number of unobserved constructs, the latent factors, cause the observed answers. A guest has a real but invisible opinion of the service, and that opinion, plus some item specific noise, produces her rating of the reception, of the staff's speed, of their knowledge of the region. Factor analysis divides each variable's variance into a common part explained by the factors and a unique part that belongs to that item alone. PCA keeps all the variance; FA only wants the shared part.

The practical consequence: when you believe your items measure constructs, as in a survey, factor analysis is the honest tool. When you just want to compress correlated behavioural variables into inputs for a cluster analysis or a regression, PCA is faster, more stable and needs fewer assumptions. On many datasets the two give similar loadings, which is why they are confused, but they diverge when the unique variances are large, and the word you use afterwards, component or factor, should reflect which one you ran.

The maths of principal components

Standardise every variable to mean zero and standard deviation one, so that a spend column in pounds does not dominate a click rate between 0 and 1. The correlation matrix $R$ of the $p$ standardised variables is then decomposed as

R = V \Lambda V^{\top}, \qquad \Lambda = \mathrm{diag}(\lambda_1, \lambda_2, \dots, \lambda_p), \qquad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p \ge 0

Here $R$ is the $p$ by $p$ correlation matrix, $V$ is a $p$ by $p$ matrix whose columns are the eigenvectors (each column holds the weights that define one component), $V^{\top}$ is its transpose, and $\Lambda$ is a diagonal matrix of the eigenvalues, sorted from largest to smallest. Each eigenvalue is the variance of one component. The $k$ th component score for a guest is a weighted sum of her standardised values:

z_k = v_k^{\top} x = v_{1k}\, x_1 + v_{2k}\, x_2 + \dots + v_{pk}\, x_p

where $x$ is her vector of $p$ standardised variables and $v_{jk}$ is the weight of variable $j$ in component $k$ . Because the trace of a correlation matrix equals $p$ (every variable contributes a variance of exactly one), the eigenvalues add up to $p$ and the share of total variance carried by component $k$ is simply

\text{share}_k = \frac{\lambda_k}{\sum_{j=1}^{p} \lambda_j} = \frac{\lambda_k}{p}

This gives the two classic rules for how many components to keep. Kaiser's rule keeps every component with an eigenvalue above one, on the argument that a component worth less than a single original variable is not compressing anything. The scree plot draws the eigenvalues in order and looks for the elbow where the curve flattens into rubble. I use both, then run a parallel analysis, which decomposes random data of the same size and keeps only components whose eigenvalues beat what randomness alone produces. When the three agree, I trust the count. When they disagree, I keep the smaller number and say so.

The factor model

Factor analysis writes each observed variable as a weighted sum of a few latent factors plus its own unique part:

x_i = \lambda_{i1} f_1 + \lambda_{i2} f_2 + \dots + \lambda_{im} f_m + e_i, \qquad \text{or in matrix form} \qquad \mathbf{x} = \Lambda \mathbf{f} + \mathbf{e}

Here $x_i$ is the standardised score on item $i$ , $f_1$ to $f_m$ are the $m$ latent factors (far fewer than the $p$ items), $\lambda_{ik}$ is the loading of item $i$ on factor $k$ , and $e_i$ is the unique part of item $i$ , the bit no factor explains. In matrix form $\mathbf{x}$ holds all $p$ items, $\Lambda$ is the $p$ by $m$ loading matrix and $\mathbf{e}$ the uniquenesses. If the factors are uncorrelated with unit variance, the model implies

R = \Lambda \Lambda^{\top} + \Psi, \qquad h_i^2 = \sum_{k=1}^{m} \lambda_{ik}^2, \qquad \psi_i = 1 - h_i^2

where $\Psi$ is a diagonal matrix of the unique variances $\psi_i$ and $h_i^2$ is the communality of item $i$ , the share of its variance that the factors explain. An item with communality 0.70 is well represented; one with 0.25 is mostly noise or measures something the factors do not cover, and should be questioned or dropped. Estimation is by maximum likelihood or principal axis factoring; both iterate until implied and observed correlations agree as closely as possible.

One thing about factor analysis surprises people the first time: the solution is not unique. Any rotation of the factor axes fits the data exactly as well. That is not a flaw, it is a freedom, and you use it to pick the rotation that is easiest to read. Varimax keeps the factors at right angles and pushes each item to load strongly on one factor and weakly on the rest. Oblimin lets the factors correlate, which is more realistic for surveys, where a guest who feels well served also tends to feel the price was fair. I start with varimax because the table is cleaner, then check oblimin to see whether the story changes. With orthogonal factors a loading is simply the correlation between item and factor: 0.81 means "very much this factor", 0.12 means "barely at all".

Seeblick Hotels: 24 items, four factors

Anna's 1,340 responses, 24 items, 24 eigenvalues adding up to 24. The first twelve:

Eigenvalues in order; the elbow after the fourth bar and the drop below 1.0 at the fifth both point to four factors.

The first component alone carries 7.9 of 24, that is 32.9 per cent of all variance, which is the "everything is fine or everything is bad" tendency you find in almost every satisfaction survey. The second carries 15.0 per cent, the third 10.0, the fourth 7.1. Together the four explain 65.0 per cent. The fifth eigenvalue drops to 0.9, below Kaiser's line, the scree flattens, and parallel analysis on random data of the same shape gives a fourth eigenvalue of about 1.25 and a fifth of about 1.19, so the fourth real factor beats chance and the fifth does not. Four it is.

Then the maximum likelihood fit with varimax rotation. Twelve of the 24 items with their rotated loadings (the other twelve follow the same pattern):

Survey item	Service	Room	Food	Value
Reception was warm and welcoming	0.81	0.12	0.09	0.15
Staff solved problems quickly	0.78	0.14	0.11	0.10
Staff knew the region well	0.72	0.08	0.13	0.06
Room was clean on arrival	0.11	0.84	0.07	0.09
Bed and sleep quality	0.09	0.79	0.05	0.12
Bathroom condition	0.15	0.76	0.08	0.11
Breakfast quality	0.12	0.10	0.83	0.14
Dinner quality	0.10	0.07	0.80	0.18
Regional products on the menu	0.18	0.05	0.69	0.09
Price felt fair for what we received	0.16	0.13	0.19	0.79
Extras were fairly priced	0.08	0.09	0.15	0.77
Would book direct again at this price	0.21	0.11	0.12	0.66

Read it row by row. "Reception was warm and welcoming" loads 0.81 on the first factor and below 0.16 on the other three. Its communality is $0.81^2 + 0.12^2 + 0.09^2 + 0.15^2 = 0.70$ , so the four factors explain 70 per cent of how guests rate the reception and the remaining 30 per cent is item specific: the particular receptionist on the particular evening. The next two rows do the same, so factor one is about people, and I call it Service. Rows four to six cluster on the second factor with loadings from 0.76 to 0.84 and nothing above 0.15 elsewhere: Room. Rows seven to nine define Food, with "regional products on the menu" weaker at 0.69, which says it is partly about food and partly about the guest's interest in the region at all. The last three rows form Value. Note the last one: "would book direct again at this price" loads 0.66 on Value but also 0.21 on Service. That mild cross loading is informative: the decision to come back is mostly about fairness and a little about how you were treated.

One item did not survive. "WiFi speed" loaded 0.31 on Room, 0.38 on Service and next to nothing elsewhere, for a communality of 0.29. It measures none of the four constructs; it is its own small thing. I dropped it from the solution and reported it separately, the right treatment for an item that matters operationally but belongs to no construct.

The structure the model found, latent causes on top and observed answers underneath:

The factor model as a picture: four latent constructs each cause three of the observed ratings shown here; the arrows run from the invisible to the visible.

From four factors to one decision

Naming factors is pleasant but it is not a decision. Anna's question had two parts: what to fix first and whether it moves direct bookings. The survey also asked "how likely are you to recommend us" on a 0 to 10 scale, so I computed each guest's four factor scores (standardised, mean zero, standard deviation one) and regressed the recommendation score on them:

Predictor	Coefficient	Standard error	Reading
Intercept	8.31	0.04	the average guest at average factor scores
Service	0.62	0.05	one standard deviation more service adds 0.62 points
Room	0.41	0.05	the smallest lever of the four
Food	0.55	0.05	matters most in the two houses with restaurants
Value	0.74	0.06	the largest lever, and it is not the room rate

The model explains 46 per cent of the variance in recommendation. Because varimax factors are uncorrelated, these coefficients do not fight each other the way 24 raw items would; you can read each one on its own. Value is the strongest lever. But look back at the loadings: the Value factor is built as much from "extras were fairly priced" as from the room rate. When we split the extras question into its parts in a follow up, parking, spa access and the minibar were the sore points, all three pricing policy decisions that cost almost nothing to change, not a discount on the room. Room, where Anna had planned to spend most of next year's budget, is the weakest of the four levers for the guests she already has. That is the whole payoff: 24 columns collapsed into four, and the four ranked against the one number the business cares about. Whether price sensitive platform guests react differently is a question for the next chapter, which is why the factor scores stay as columns in the guest table instead of dying with the report.

The Gift Bow: 18 behavioural variables, three components

The second example needs no survey at all. The Gift Bow is my Solidus demo hamper shop, and for each of roughly 6,200 customers with an order in the last 14 months I can compute 18 behavioural variables: orders, spend, average order value, distinct categories, months active, days since last order, share of orders with a gift message, share shipped to an address other than the billing address, December share, whether a company name appears on the invoice, voucher share, average discount depth, click rate on offer emails, days between an offer email and the next order, and a few more. Chapter 14 wants to cluster these customers. Feeding 18 correlated columns into k means would be a mistake: orders and spend correlate at 0.91, so the algorithm would count the same information twice and the "segments" would be spend bands in disguise.

This is a PCA job. I have no theory that a latent "gifting orientation" causes the gift message share; I simply want the fewest uncorrelated columns that keep most of the information. Standardise, decompose:

The first ten eigenvalues of the 18 behavioural variables; the fourth sits just above 1.0 and fails the parallel analysis, so three components are kept.

The fourth eigenvalue, 1.1, passes Kaiser's rule and fails parallel analysis (random data of this shape gives a fourth eigenvalue around 1.15). I keep three, which carry 10.7 of 18, or 59 per cent of the variance. The rotated loadings are easy to name:

Component	Share of variance	Variables with the largest loadings
1 Volume	28.3 per cent	orders 0.88, spend 0.86, distinct categories 0.74, months active 0.71, days since last order minus 0.69
2 Gifting orientation	18.9 per cent	gift message share 0.83, ship to other address 0.81, December share 0.72, company on invoice 0.58
3 Deal sensitivity	12.2 per cent	voucher share 0.85, average discount depth 0.79, offer click rate 0.66, days from offer to order minus 0.61

The sign matters here. "Days since last order" loads minus 0.69 on Volume because a customer who buys a lot was, almost by definition, here recently; a negative loading says the variable moves against the component. "Days from offer email to order" loads negatively on Deal sensitivity for the same reason: the deal sensitive customer orders the day the voucher lands. Component two is where the shop's real structure sits. It separates the gift buyer, sending hampers to other people's addresses with a message card, from the self buyer with the same order count and spend who ships everything home. The component finds this without me telling it what a gift buyer is, a good sign that the distinction lives in the data and not just in my head. Three scores per customer now go into chapter 14 as clustering inputs, and because they are uncorrelated, k means treats each dimension fairly instead of triple counting spend. The wider argument for why behaviour beats demographics as a basis for segments is in Segmentation is not demographics; this is the mechanical step that makes it possible.

Running it yourself

You need a wide table, one row per respondent or customer, one column per variable, with few gaps. Factor analysis on survey data wants five to ten responses per item and a floor of around 200 in total; 24 items with 1,340 answers is comfortable, 24 items with 90 answers is a coin toss. Behavioural variables are usually heavily skewed (a few corporate buyers spend fifty times the median), so log counts and amounts before standardising, otherwise the first component is just "the whales versus everybody else".

In Python the PCA side is a dozen lines:

import numpy as np, pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

X = pd.read_csv("hamper_behaviour.csv").set_index("customer_id")
X[["orders", "spend", "days_since_last"]] = np.log1p(X[["orders", "spend", "days_since_last"]])
Z = StandardScaler().fit_transform(X)            # PCA on the correlation matrix
pca = PCA().fit(Z)
eig = pca.explained_variance_                    # the eigenvalues, largest first
print(np.round(eig, 2), np.round(eig.cumsum() / eig.sum(), 3))
scores = PCA(n_components=3).fit_transform(Z)    # three columns for chapter 14

For the survey I prefer R and the psych package, which does parallel analysis, rotation and reliability in one place:

library(psych)
items <- survey[, 5:28]                         # the 24 rating items
fa.parallel(items, fa = "fa")                   # how many factors beat random data
fit <- fa(items, nfactors = 4, rotate = "varimax", fm = "ml")
print(fit$loadings, cutoff = 0.30)              # hide the small loadings
fit$communality                                 # anything under 0.4 needs a look
scores <- fit$scores                            # one row per guest, four columns

Time: a survey factor analysis is a week of work including cleaning, rotation checks and write up, and most of the week is the cleaning. A behavioural PCA is two days once the customer table exists, and the customer table is where the real effort goes.

Before I trust a solution I run five checks. The Kaiser Meyer Olkin measure of sampling adequacy should be above 0.7; below 0.6 the variables do not share enough variance to factor at all. Cronbach's alpha for the items of each factor should be above 0.7, which says the items agree with each other. Parallel analysis should confirm the count. I split the sample in half at random, fit both halves and compare the loadings; if the halves disagree about which items belong where, the structure is not stable and I report fewer factors. And I check whether an oblique rotation changes the story; if oblimin says two factors correlate at 0.7, they may be one factor, and I should say so rather than present a tidy four when the data prefers three.

Pitfalls

The naming fallacy. The moment you call a factor "Value" you begin to believe the data measured value. It did not. It measured three questions that a rotation grouped together, and your name is a hypothesis about why. I once saw a factor labelled "brand love" built entirely from items about delivery speed. Name factors from their items, keep the items next to the name in the report, and treat the name as a working title.

Kaiser's rule on autopilot. Eigenvalue above one overextracts on large item sets and underextracts on small ones. With 60 items it will hand you nine factors, half of them a single item talking to itself. Parallel analysis and the scree plot overrule it, and interpretability overrules all three: a factor you cannot name from its items is not a finding.

Forgetting to standardise. PCA on raw variables finds the direction of largest variance in whatever units you happened to use. Spend in pounds has a variance in the hundreds of thousands, a click rate has a variance of 0.02, and the first component becomes spend with a rounding error. Standardisation is not optional.

The agreeable guest. Satisfaction items on a 1 to 10 scale have a ceiling, and some people give everything an 8 out of politeness. That produces a large first unrotated component that is really a mood factor, not a construct. Rotation helps, reverse worded items help more, and reporting that component as "overall satisfaction" without saying so is a small dishonesty that adds up.

Reifying components. A component summarises a particular dataset in a particular season. Rerun the hamper shop analysis next year with new products and the gifting component may absorb the corporate variables or split in two. Components are tools for the decision in front of you, not permanent facts about your customers, and dashboards built on them should carry a date.

How I do this for clients

The deliverable depends on which of the two problems you have, and the first hour is spent finding out. If you have a survey, or want one, I need the item wording, the raw responses one row per respondent and, ideally, one business outcome per respondent, such as whether they rebooked or renewed. If you have a customer database, I need the order and event history so that I can build the behavioural table myself, because the variables already sitting in the CRM are rarely the right ones.

It starts with a free workshop, a couple of hours in which we agree the decision the analysis has to support and what the output must look like for you to act on it. Then two weeks of real work on my account, before you commit to anything. For a survey that means the cleaned data, the factor solution with loadings and reliabilities, factor scores written back to your respondent table, and a two page memo that ranks the factors against your outcome with the coefficient, its error margin and the risk I see in acting on it. For a database it means the behavioural table, the component solution and the scores stored as columns you own, ready for the segmentation that follows, plus a memo saying what each component means in plain words and what it does not.

Everything I build runs on your systems and belongs to you; there is no proprietary model to rent. The data science page describes the range of work, and the pricing page describes how I charge: a fixed fee for the analysis and a commission based option when the result becomes a growth programme. One accountable person, honest numbers with their uncertainty attached, no vanity metrics.

Questions to ask before you act on a factor solution

How many respondents or customers per variable went into this, and were there enough to trust the loadings?
Which rule decided the number of factors, and would parallel analysis or the scree plot have given a different count?
Were the variables standardised or logged before the decomposition, and which ones dominated if not?
Which rotation was used, was it chosen before the loadings were seen, and does an oblique rotation change the story?
Do the factor names follow from the items in the table, or from what we hoped to find?
Have the factor scores been linked to a business outcome, and how much of that outcome do they explain?
Are the scores stored where the next analysis can use them, with a date on them?

This series is inspired by Mike Grigsby's Marketing Analytics (Kogan Page). The explanations, examples and numbers here are my own.