Modelling demand and price elasticity with regression

The hamper that would not tell me its price

The Afternoon Tea Hamper is the quiet workhorse of The Gift Bow, my Solidus demo shop at hampers.keferboeck.com. It sells all year and it has sat at £49 for so long that the price feels like a fact of nature. Last spring I wanted to know whether it should be £54. My marketing instinct said the opposite: drop it to £45, sell more, win the gift buyer who is comparing five tabs. My finance instinct said raise it, the margin is thin once the courier has been paid. Both instincts are guesses about the same unknown number: how many hampers I lose for every pound I add.

The good news is that the shop had already run the experiment for me, badly and by accident. Over the previous year the price had moved seven times, between £39 in a January clearance and £59 in the run up to Christmas. There had been eleven promotion weeks with a free delivery code, six weeks of Christmas madness, and a competitor whose comparable hamper wandered between £52 and £58. Fifty two weeks of price, units and context: exactly the raw material a regression needs. This chapter turns it into one number you can defend in a meeting, the price elasticity of demand. The numbers are illustrative, the method is the one I run for real.

Where this sits in the series

This is chapter 4 of 21 and the first of Part two, the dependent variable techniques, which run from here to chapter 10. Part one asked what analytics is for and ended with What is an insight, and why most reports do not contain one, where I argued that an insight has to change a decision. This chapter is the first method that reliably produces one. The next chapter, Polynomial distributed lags: how long does marketing keep working?, keeps the same regression machinery and adds time, because an advert seen this week still sells hampers next month.

Two families of technique, and the question that separates them

Nearly every method in marketing analytics belongs to one of two families, and the sorting question is: is there a variable you are trying to explain?

If yes, you are in the dependent variable family. Units sold, revenue, churn, whether a trial converts, how many nights a guest books: something on the left hand side of an equation that you believe is caused, at least in part, by the things on the right. Regression, logistic regression, survival models and their relatives live here, and so does this whole part of the series.

If no, you are in the interrelationship family: no target, only structure, such as which customers resemble each other or which products share a basket. Segmentation and factor analysis live there, and we get to them in Part three.

Within the first family there is a second distinction. Some equations are deterministic: revenue equals price times units, always and exactly, nothing to estimate. Others are probabilistic: units this week depend on price, promotion and season, plus something we did not measure. The something is the error term, and putting it in the equation is the honest admission that the model is a simplification. Every technique in this part of the series estimates a probabilistic equation and then says how wrong it might be. A demand forecast without an error margin has either hidden the uncertainty or pretended a deterministic identity was a model.

Ordinary least squares, the workhorse

The demand model for one hamper, in its plainest linear form, looks like this:

$$Q_t = \beta_0 + \beta_1 P_t + \beta_2 D_t + \beta_3 S_t + \beta_4 C_t + \varepsilon_t$$

$Q_t$ is the number of hampers sold in week $t$. $P_t$ is our price that week. $D_t$ is a promotion dummy, 1 if a discount or free delivery code was live and 0 otherwise. $S_t$ is a season dummy, 1 in the six Christmas weeks. $C_t$ is the competitor's price. The betas are the unknowns we want: $\beta_0$ is the baseline, $\beta_1$ is the change in weekly units for one extra pound of price, and so on. $\varepsilon_t$ is the error term, everything about week $t$ the model does not know.

Ordinary least squares, OLS, picks the betas that bring the predicted units as close as possible to the actual units across all 52 weeks, where "close" means the sum of squared misses:

$$\min_{\beta_0,\ldots,\beta_4} \; \sum_{t=1}^{n} \left( Q_t - \hat{Q}_t \right)^2$$

$\hat{Q}_t$ is the model's prediction for week $t$ and $n$ is the number of weeks. Squaring the misses stops positive and negative misses cancelling and punishes one huge miss more than several small ones. Your software finds the solution in a millisecond, and the method has been the default for two hundred years because it is simple, transparent and, when its assumptions hold, the best linear estimator available.

What comes back is a table, and reading that table is the actual skill. The coefficient is the estimate itself: minus 1.95 would mean each extra pound costs about two hampers a week. The standard error is the uncertainty around it, the typical distance between this estimate and the one a different year of the same shop would give. The t statistic is the coefficient divided by its standard error; with 50 odd weeks an absolute t above about 2 means the sign is unlikely to be luck. R squared is how much of the week to week variation in units the model explains:

$$R^2 = 1 - \frac{\sum_{t} \left( Q_t - \hat{Q}_t \right)^2}{\sum_{t} \left( Q_t - \bar{Q} \right)^2}$$

The numerator is what the model leaves unexplained, the denominator is the total variation around the average $\bar{Q}$, and the ratio is the fraction still missing. An R squared of 0.81 means the model accounts for 81 percent of the ups and downs. It does not mean the model is right; a high R squared with a nonsense coefficient is worse than a low one with a sensible coefficient. Finally the residuals, the misses week by week. Plot them against time and against price. Random noise is good. Trends, clusters or a fan shape mean the model is missing something and the standard errors are lying to you.

From a coefficient to an elasticity

A coefficient of minus 1.95 hampers per pound is useful for this hamper and useless for comparing with a €120 Werkbank plan or a €190 hotel room. Elasticity fixes that by putting both sides in percentages: the percentage change in quantity for a one percent change in price.

$$\eta = \frac{\partial Q}{\partial P} \cdot \frac{P}{Q}$$

$\eta$ is the elasticity, $\partial Q / \partial P$ is the slope of demand with respect to price, which in the linear model is just $\beta_1$, and $P/Q$ rescales it from pounds and hampers into percentages. Because $P/Q$ changes along the demand line, a linear model's elasticity differs at every price, so by convention you evaluate it at the means. With $\beta_1$ of minus 1.95, an average price of £49 and average weekly sales of 60 hampers, that is minus 1.95 times 49 divided by 60, which is minus 1.59. A one percent price rise loses about 1.6 percent of volume. Demand is elastic, more than one in absolute value, and that single fact drives the whole pricing discussion that follows.

If you want the intuition behind the number without the algebra, I wrote it up in Price elasticity in plain English; this chapter is the part where we actually estimate it.

The log log model: one coefficient, one elasticity

The linear model gives you an elasticity that depends on where you stand. For pricing work I usually reach for a different functional form first, because it makes the elasticity the thing you estimate directly. Take logarithms of quantity and of every price:

$$\ln Q_t = \alpha + \eta \ln P_t + \gamma_1 D_t + \gamma_2 S_t + \theta \ln C_t + \varepsilon_t$$

$\ln$ is the natural logarithm, $\alpha$ is the intercept, and $\eta$, the coefficient on log price, is now the own price elasticity itself, constant at every price level. The trick is that a small change in the log of a variable is a percentage change, so a regression of log on log is a regression of percentages on percentages. $\theta$ is the cross price elasticity, the percentage change in our units for a one percent change in the competitor's price, positive for a substitute. The dummies $D_t$ and $S_t$ stay as they were, so $\gamma_1$ and $\gamma_2$ are approximately percentage lifts; the exact lift is $e^{\gamma} - 1$.

The log log model has a cost, which I come back to under pitfalls: it assumes constant elasticity, and demand rarely obliges across the whole price range. As a first estimate on a year of shop data it is still hard to beat.

The worked example: 52 weeks of the Afternoon Tea Hamper

Here is the output of the log log model on the year of Gift Bow data: 52 weekly observations, price between £39 and £59 at seven distinct levels, eleven promotion weeks, six Christmas weeks from mid November, competitor price scraped every Monday.

Term	Coefficient	Std error	t statistic
Intercept	8.31	1.92	4.3
ln price (own)	minus 1.61	0.24	minus 6.7
Promotion week (0/1)	0.28	0.09	3.1
Christmas week (0/1)	0.74	0.12	6.2
ln competitor price	0.52	0.31	1.7

n = 52 weeks, R squared 0.81, adjusted R squared 0.79, residual standard error 0.14 in log units, roughly a 14 percent typical miss.

Reading it line by line. The intercept is the log of units at a price of £1 with a competitor at £1, so it has no business meaning and I ignore it. The own price coefficient is the headline: elasticity of minus 1.61, standard error 0.24, so the plausible range is roughly minus 1.1 to minus 2.1 and the t statistic of minus 6.7 says the sign is not in doubt. Whether it is minus 1.2 or minus 2.0 matters a great deal for pricing, and a year of data does not pin it down tighter than that.

Promotion at 0.28 means a free delivery week lifts units by about 32 percent, since $e^{0.28}$ is 1.32, holding price constant; that is the code's effect over and above the price cut it usually comes with, which is why price belongs in the same model. Christmas at 0.74 is a 110 percent lift, demand doubles, and its t of 6.2 makes it the most certain thing in the table. The competitor coefficient needs care: 0.52 says a ten percent rise in their price gives us about five percent more units, the right sign for a substitute, but the t of 1.7 means I would not bet the house on the size. Competitor prices barely moved in the year, so there is little information there, and the standard error says so honestly.

The chart shows the same model as a demand curve: observed average weekly units at each price point as bars, the model's prediction as the line, ordinary weeks only.

Observed average weekly units (bars) against the fitted log log demand curve (line), baseline weeks only. Illustrative data.

The curve is steeper at the low end and flatter at the high end. That is what constant elasticity looks like in units: the same percentage loss per percentage rise means fewer absolute hampers lost as the price climbs.

Revenue, contribution and the price that pays

Now the part the meeting actually wants. Revenue is price times units, so the effect of a small price rise on revenue is:

$$\frac{dR}{dP} = Q + P \frac{dQ}{dP} = Q \left( 1 + \eta \right)$$

$R$ is revenue and the rest is as before. Revenue rises with price while $1 + \eta$ is positive, that is while demand is inelastic, and falls while it is negative. Revenue peaks exactly where $\eta = -1$. With an elasticity of minus 1.6 that condition tells us we are past the revenue peak: every price rise loses revenue, every price cut gains some. Read carelessly, the model says cut the price.

Read carefully, it says no such thing, for two reasons. First, a constant elasticity model of minus 1.6 has no revenue peak anywhere: it says cut, and keep cutting, to a price of zero, which is a property of the functional form and not of hampers. The linear version has a peak at $P^* = -\beta_0 / (2 \beta_1)$, which with my linear coefficients lands near £40. Two sensible models, two revenue maximising prices, and the honest conclusion is that a revenue peak estimated from a year of data is soft.

Second, and more important, nobody should maximise revenue. A hamper costs me about £19 to sell: goods, box, ribbon and courier. Contribution pays the bills, and the contribution maximising price under constant elasticity is a clean rule:

$$P^* = c \cdot \frac{\eta}{1 + \eta}$$

$c$ is the variable cost per unit and $\eta$ the elasticity. With £19 and minus 1.61 that is 19 times 1.61 divided by 0.61, about £50. Here is the same story as a price ladder, using the fitted units from the model.

Price	Model units per week	Revenue	Contribution at £19 cost
£39	87	£3,393	£1,740
£42	77	£3,234	£1,771
£45	69	£3,105	£1,794
£49	60	£2,940	£1,800
£52	55	£2,860	£1,815
£55	50	£2,750	£1,800
£59	45	£2,655	£1,800

Revenue falls all the way down the ladder, as the elasticity said it would. Contribution is almost flat from £45 to £59, with a shallow top around £50 to £52. That flatness is the real finding. The shop is near the top of a broad hill, and the question of £49 versus £54 is worth about £15 a week either way, well inside the model's error. The marketing instinct to cut to £45 would have cost contribution for the pleasure of more parcels. The finance instinct to go to £54 is defensible and not urgent. For why £49 and £54 are not the same to a gift buyer even when the maths says they nearly are, pricing psychology covers the part regression cannot see.

Elasticity is not one number: corporate versus private buyers

The minus 1.6 is an average over everyone who bought the hamper, and averages hide the useful part. The Gift Bow has three recognisable groups: corporate buyers ordering five or more on an invoice, private gift buyers, and private self buyers. Splitting the weekly data by segment and refitting the same model gives this:

Segment	Share of units	Elasticity	Std error
Corporate orders	31%	minus 0.7	0.30
Private gift buyers	54%	minus 1.9	0.28
Private self buyers	15%	minus 2.4	0.55
All buyers	100%	minus 1.6	0.24

Absolute own price elasticity by buyer segment from the segment level log log models. Illustrative data.

Corporate demand is inelastic. A purchasing manager sending forty hampers to clients is spending a budget approved in September, the sign off is the same at £49 as at £54, and the alternative is not a lower priced hamper but the effort of finding a new supplier. Private self buyers are the opposite, and the wide standard error says there are not many of them to learn from. Gift buyers sit in between and dominate the volume.

The practical consequence is that one list price is the wrong instrument. A corporate price list with volume tiers above the web price, and a web price tuned for the gift buyer, will beat any single number, and the regression is what tells you the gap is real rather than a hunch. Cross price elasticity splits the same way: near zero for corporate buyers, who are not comparing tabs, larger for self buyers, who are. Take that further, letting price move by segment and by time on the model's say so, and you are in the territory of dynamic pricing, which deserves its own playbook.

The test that settles it

Everything above is observational. I set the prices in the data, for reasons, and some of those reasons were related to demand, which is the first pitfall below. So the model is a hypothesis with a number attached, and the proof is a field test with a control group.

In week 37, an ordinary October week with no promotion, half of web sessions, chosen at random, saw the hamper at £54 and the other half at £49. The model predicted private buyers would drop by about 1.9 times 10 percent, roughly 18 percent. The control group bought 61 hampers per thousand sessions, the test group 49, a fall of 20 percent, with an interval from about minus 8 to minus 31 percent at these volumes. Revenue per thousand sessions was £2,989 in control against £2,646 in test; contribution £1,830 against £1,715. For the web shop, £49 won, and the model had called it. On the corporate side I quoted £54 to half of that month's enquiries and £49 to the rest: 11 of 14 accepted the higher price, 12 of 15 the lower, too few to prove anything and entirely consistent with an elasticity near minus 0.7.

Test versus control is the proof because it removes the confounding at source: same week, same weather, same competitor price, same distance to Christmas, only the price differed. The regression told me where to look and how big an effect to expect; the test told me it was real. If you can only afford one of the two, run the test. If you can afford both, the model makes the test smaller and sharper.

How to run this yourself

You need one row per week for one product: units sold, average realised price, a promotion flag, season flags, and any competitor price you can collect. Building that table is most of the work. In a Solidus shop it is a query like this:

select date_trunc('week', o.completed_at)::date as week,
       sum(li.quantity) as units,
       sum(li.price * li.quantity) / sum(li.quantity) as price,
       max(case when o.promo_total <> 0 then 1 else 0 end) as promo
from spree_line_items li
join spree_orders o on o.id = li.order_id
join spree_variants v on v.id = li.variant_id
where v.sku = 'AFTERNOON-TEA' and o.state = 'complete'
group by 1 order by 1;

Add the Christmas flag and the competitor price, then fit the model. In Python with statsmodels:

import numpy as np
import statsmodels.formula.api as smf

df["ln_units"] = np.log(df["units"])
df["ln_price"] = np.log(df["price"])
df["ln_comp"] = np.log(df["competitor_price"])

model = smf.ols("ln_units ~ ln_price + promo + christmas + ln_comp", data=df)
fit = model.fit(cov_type="HAC", cov_kwds={"maxlags": 4})
print(fit.summary())
print("elasticity:", round(fit.params["ln_price"], 2))

The HAC option gives standard errors that survive the autocorrelation of weekly data. R users get the same from lm plus the sandwich package.

Time: a day to build a clean weekly table, an hour to fit and plot, and about a week of arguing about what the coefficients mean. Before you trust the result, five checks. Count the distinct price levels; fewer than five and the elasticity is a guess with decimal places. Plot residuals against time and against fitted values and look for structure. Check the correlation between the promotion flag and the season flag; above about 0.6 they are fighting over the same weeks. Refit on the first 40 weeks, predict the last 12, and see whether the misses are the size the residual standard error promised. And read the sign of every coefficient before the size, because a positive own price coefficient is not a discovery, it is a data problem.

The route from an order database to a defensible price decision. The test is not optional.

Pitfalls

Price is not set at random. This is the big one, and it has a name, endogeneity. If you raise prices when demand is strong and cut them when it is weak, price and the error term move together, and OLS will report an elasticity that is too small, or even the wrong sign. Season dummies absorb some of that; the rest needs price changes made for reasons unrelated to demand, or a randomised test. When a client's history shows every price rise in November, I say so before I fit anything.

Promotions live in the season. If every promotion ran in December, the model cannot separate the code from the Christmas lift, and the standard errors on both will balloon. The cure is boring: run at least a few promotions in ordinary months, so that the data contains promotion weeks without Christmas and Christmas weeks without promotion.

Too little variation. A price that moved twice in a year gives you two points and a line through them. The standard error will be enormous, but people read the coefficient and skip the rest. If you want to know your elasticity, you have to move the price, deliberately and in both directions.

Extrapolation. The data covered £39 to £59. It says nothing about £29 or £79, and the constant elasticity form will happily produce a number for either. An elasticity is valid over the range of prices you actually charged, and no further.

Constant elasticity is an assumption. The log log model forces the same percentage response everywhere. Real demand often has a kink, a threshold like £50 where behaviour changes. Add a squared log price term or fit a piecewise model; if the fit improves, quote different elasticities for different ranges.

How I do this for clients

The deliverable is a pricing decision with a number and a risk on it, not a model. It starts with the free workshop: half a day to work out which products have moved in price often enough to say anything, what the cost per unit really is, and what decision the elasticity would change. If the answer is "none", I say so and we stop there.

Then I spend two weeks on it before you commit to anything. I need order lines with realised prices, whatever promotion history exists, and any competitor pricing you have or can start collecting. From that I build the weekly table, fit the linear and log log models, split by segment where the data allows, and design a test versus control week for the product that matters most. You get three things: the model itself, in a notebook you own and can rerun; a price ladder like the one above with revenue and contribution at every level, with the error range shown rather than hidden; and a one page decision memo that says what price to test, what the model expects to happen, and what would make me wrong. If you want the elasticity kept alive, it becomes one dashboard panel that answers one question: are we still on the flat part of the hill?

This sits under my data science work. It costs what two weeks of one accountable person costs, spelled out on the pricing page, and for shops where the pricing change flows straight into revenue I am open to a commission based arrangement on the uplift. You own everything I build, including the query, the model and the memo.

Questions to ask before you move a price

• How many distinct price levels does our history contain for this product, and over what range?
• Is the elasticity quoted with a standard error or confidence interval, and how wide is it?
• Were the price changes in the data made for reasons unrelated to demand, and if not, how was that handled?
• Are promotions and season in the model separately, and how correlated are they?
• Is the recommended price a revenue maximum or a contribution maximum, and what cost per unit was used?
• Does the elasticity differ by customer segment, and does our price structure reflect that?
• What test versus control result would make us abandon the recommendation?

This series is inspired by Mike Grigsby's Marketing Analytics (Kogan Page). The explanations, examples and numbers here are my own.