Why I moved from growth hacking to data driven eCommerce growth

I swapped clever tactics for numbers I could stand behind

For a while I did growth hacking on informational websites: blogs, guides, content sites whose job is to inform or entertain. Over time I moved away from that, towards transactional websites, meaning eCommerce stores and web apps that sell subscriptions.

The reason was the goal. An informational site wants traffic and engagement, and I could never say with a straight face what a page view was worth. A shop wants sales, and a subscription app wants subscribers who stay. Those goals can be counted, priced and tested. Through my work as a data scientist I came to see data as the thing that makes growing a transactional business manageable, because it lets me decide from what customers actually do instead of from what I assume they will do.

This article is the honest version of that move. First, what growth hacking was meant to be, because the popular picture is not quite fair to it. Then the four things that change when growth work moves from tactics to measurement: margin instead of revenue, incrementality instead of attribution, cohorts instead of totals, and experiments with rules instead of experiments with hope. Each comes with a worked example and links to the longer articles that go deeper.

All the examples use one invented business. Corrib Coffee is an online coffee roaster in Galway, run by an equally invented owner called Sam, selling beans, grinders and a monthly subscription at corribcoffee.example. Every number is made up so I can show every step. Nobody real.

Growth hacking began as measurement, and the hype turned it into tricks

The term comes from Sean Ellis. In July 2010 he wrote a blog post called "Find a Growth Hacker for Your Startup" and defined the role in one line: "A growth hacker is a person whose true north is growth." He went on: "Everything they do is scrutinized by its potential impact on scalable growth."¹

Two years later Andrew Chen made the term famous with an essay titled "Growth Hacker is the new VP Marketing". His central example was Airbnb letting hosts post their listings to Craigslist, an integration built without any public Craigslist interface, and tracked with unique links.² The point of the story was engineering plus measurement, not a slogan.

In 2017 Ellis and Morgan Brown wrote the method down in the book Hacking Growth. Its publisher describes the method as cross functional teams running continuous tests and iterating on the results.³

So the original idea was close to what I do now. Pick the one number that matters, test ideas against it, keep what moves it. What spread across the internet was something else: lists of tactics stripped of the measurement that made them work. I am in no position to look down on that. This very site has an article called 53 growth hacks you can implement this afternoon. The tactics in it are fine. A tactic without a number to judge it by is just a guess with a nice name.

If you want the longer story of the role and how it got diluted, I wrote it up in what is a growth hacker.

Worked example: one tactic, judged two ways

Sam reads that a pop up offering 10% off the first order is a proven growth hack. He switches it on.

Judged the tactics way, it is a hit. Sign ups to the email list go from 40 a week to 260 a week. That is a gain of 220 sign ups, or 550%, because $\frac{260 - 40}{40} \times 100 = 550$.

Judged the measurement way, the question is different: did the business make more money? That needs three numbers the sign up counter does not show. How many of the new sign ups would have bought anyway, at full price. What the 10% costs on those orders. And whether the discounted customers come back. The rest of this article is how to answer each one.

I left informational sites because their goal stopped at the click

Here are the problems I kept running into with growth work on informational websites, as I described them when I first wrote this piece.

The goals were undefined. Success meant traffic and engagement, and those split into views, clicks and time on page, none of which reliably tied to business value. There was no sale or subscription at the end, only engagement, which is too vague to decide from.

Behaviour was hard to predict. I could see that someone read an article on how to start an online store, but not why, and not whether that reader ever went on to buy anything from anyone. The outcome happened somewhere I could not see.

And the feedback loop was weak. You can try new headlines, new formats and new social campaigns, but if you cannot link any of them to an outcome with a price on it, you cannot tell which ones to scale.

A transactional site fixes the first problem by design. Every visit either ends in an order or it does not, and every order has a value. Once the outcome is countable, a rate can be computed, and a rate can be compared, tested and priced.

$$\text{conversion rate} = \frac{\text{orders}}{\text{visits}}$$

Here $\text{orders}$ is the number of completed purchases in a period, $\text{visits}$ is the number of sessions on the site in the same period, and the conversion rate is the share of visits that end in an order. Rates and their denominators are explained in Foundations: ratios and rates.

Worked example: the same traffic, two kinds of site

Picture two sites in Galway, each with 50,000 visits in November.

The first is a coffee blog. It reports 50,000 visits and an average of three minutes on the page. What were those visits worth? There is no number to multiply by. Any answer is a story.

The second is Corrib Coffee. Of its 50,000 visits, 1,000 ended in an order:

$$\text{conversion rate} = \frac{1{,}000}{50{,}000} = 0.02 = 2\%$$

The orders averaged €32, so November revenue was $1{,}000 \times 32 = €32{,}000$. Now every change Sam makes can be judged in euro. That is the whole reason I moved. Not that one kind of site is better, but that one kind of site lets me prove what my work did.

Revenue is the wrong scoreboard, margin is the right one

The first thing I measured on transactional sites was return on ad spend, ROAS: revenue divided by ad spend. In the original version of this article I even showed a Python script that merged ad data with sales data, computed ROAS, cost per acquisition and conversion rate for each campaign, and flagged every campaign with a ROAS below 3. I did build systems like that, and the merge part is still the right start.

The flag was the weak part. A fixed ROAS threshold treats a euro of revenue on a low margin grinder the same as a euro of revenue on high margin coffee beans. They are not the same euro. What a business keeps is the margin, so the honest scoreboard is contribution: margin earned minus what the ads cost.

$$C = R \times m - S$$

Here $C$ is the contribution of a campaign in euro, $R$ is the revenue it brought in, $m$ is the margin rate (the share of each euro of revenue left after product cost, packaging, payment fees and an allowance for returns), and $S$ is the ad spend. It is a rate times a total, as in Foundations: ratios and rates. Setting $C = 0$ gives the break even ROAS:

$$\text{ROAS}_{\text{break even}} = \frac{1}{m}$$

Here $\text{ROAS}_{\text{break even}}$ is the revenue per euro of spend at which the campaign exactly pays for itself, and $m$ is the same margin rate as before. Both rest on Foundations: ratios and rates, and the habit of judging the next euro rather than the average one is in Foundations: marginal versus average.

Worked example: the campaign that looked better and lost money

Sam runs two Google Shopping campaigns in October, each with €2,000 of spend. Assume, for this example, a margin rate of 15% on grinders (bought in from a supplier) and 55% on beans (roasted in house). Those margins are the example's assumption, typical of the gap between reselling hardware and making your own product.

The grinder campaign brings in €10,000. Its ROAS is $\frac{10{,}000}{2{,}000} = 5.0$ and its contribution is $C = 10{,}000 \times 0.15 - 2{,}000 = 1{,}500 - 2{,}000 = -€500$.

The beans campaign brings in €7,000. Its ROAS is $\frac{7{,}000}{2{,}000} = 3.5$ and its contribution is $C = 7{,}000 \times 0.55 - 2{,}000 = 3{,}850 - 2{,}000 = €1{,}850$.

The break even lines tell the same story faster. Grinders need $\frac{1}{0.15} = 6.67$, beans need only $\frac{1}{0.55} = 1.82$. The grinder campaign, with the better looking ROAS of 5.0, is below its line and loses €500. The beans campaign, at 3.5, is far above its line and earns €1,850.

My old script, with its single threshold of 3, would have passed both. Sam would have kept scaling the one that loses money. The fix is small: put the margin per product into the data and flag against each product's own break even line. Ad platforms can bid on margin directly if you report it as the conversion value, which I cover in auto bidding on your own data and, for Performance Max, in black box campaigns.

Attributed is not the same as caused, so measure incrementality

The revenue in that example came from the ad platform's report. That report counts the orders from people who clicked an ad. It does not tell you how many of those people would have bought anyway.

That gap has a name: incrementality. An incremental order is one that happened because of the ad and would not have happened without it. The only clean way to measure it is a holdout: show the ads to one randomly chosen group, withhold them from another, and compare. Google's own Conversion Lift studies work exactly this way, with a control group of "people who don't see your ads", and they report the incremental conversions the campaigns drove.⁴

The gap can be enormous. In a set of large field experiments at eBay, Tom Blake, Chris Nosko and Steven Tadelis found that the returns from paid search were only a fraction of what the usual, non randomised estimates suggested, and that ads on eBay's own brand keywords showed no measurable benefit in the short run.⁵ People searching for a brand by name were mostly going to arrive anyway.

$$\Delta = c_T - c_C \times \frac{n_T}{n_C}$$

Here $\Delta$ is the number of incremental orders, $c_T$ is the number of orders from the treatment group (the people who could see the ads), $c_C$ is the number of orders from the control group (the people who could not), and $n_T$ and $n_C$ are the sizes of the two groups. The fraction scales the control group to the size of the treatment group, so the two counts are comparable. Whether the difference is real or noise is a two proportion test, explained in Foundations: hypothesis tests, and the range it could plausibly be in is a confidence interval.

Worked example: Sam's brand search campaign

Sam's brand campaign bids on "corrib coffee". Over four weeks the platform attributes 450 orders to it. At €32 an order that is €14,400 of revenue for €1,500 of spend, a ROAS of 9.6. It looks like the best campaign in the account.

Sam runs a holdout. He splits 20,000 people at random into two groups of 10,000. The treatment group can see the brand ads, the control group cannot. Over the same four weeks:

• the treatment group places 520 orders, a conversion rate of 5.2%
• the control group places 480 orders, a conversion rate of 4.8%

Step one, the incremental orders. The groups are the same size, so the fraction is 1:

$$\Delta = 520 - 480 \times \frac{10{,}000}{10{,}000} = 520 - 480 = 40$$

Step two, is 40 more than noise? The pooled conversion rate is $\frac{520 + 480}{20{,}000} = 0.05$. The standard error of the difference is

$$SE = \sqrt{0.05 \times 0.95 \times \left(\frac{1}{10{,}000} + \frac{1}{10{,}000}\right)} = \sqrt{0.0475 \times 0.0002} = 0.00308$$

and the test statistic is $z = \frac{0.052 - 0.048}{0.00308} = 1.30$. That is below 1.96, so at the usual 5% level the test cannot tell the 40 orders apart from zero.

Step three, the plausible range. The 95% interval on the difference is $0.004 \pm 1.96 \times 0.00308$, which is $-0.0020$ to $0.0100$. Per 10,000 people, that is somewhere between 20 fewer orders and 100 more.

Step four, the money. Take the central estimate of 40 incremental orders. They are worth $40 \times 32 = €1{,}280$ of revenue, and at the 55% margin on beans, $1{,}280 \times 0.55 = €704$ of margin, against €1,500 of spend. The campaign loses about €796. Even the most generous end of the interval, 100 orders, gives $100 \times 32 \times 0.55 = €1{,}760$ of margin, barely above the spend. And none of the plausible range comes close to the 450 orders the report claimed.

The vertical axis is the number of orders in the four week test. The first bar is what the ad platform attributes to the campaign, the second is the incremental orders the holdout measured, and the third is the most the test allows at 95% confidence. The takeaway: most of what the report credits to brand search would have happened without it. The campaign is not necessarily worthless (it may protect the brand name from competitors bidding on it), but that is a separate question with its own price.

The longer treatment, including what to do when you cannot run a clean holdout, is in causal inference in marketing. How the platform numbers, analytics and the shop database disagree with each other is in cross language attribution.

Totals hide decline, so read growth in cohorts

A shop's monthly order count mixes new customers with returning ones. It can go up while the business gets worse, if a burst of new customers hides the fact that fewer of them come back.

The cure is a cohort. Google's own definition is plain: "A cohort is a group of users who share a common characteristic", usually the date they first arrived or first bought.⁶ You follow each group separately and ask what share is still buying a month, two months, six months later.

The value of a cohort is its customers' lifetime value, built from margin and retention:

$$CLV = M \times \frac{r}{1 + d - r}$$

Here $CLV$ is the customer lifetime value in euro, $M$ is the margin a customer earns the business per year, $r$ is the retention rate (the share of customers still buying a year later), and $d$ is the discount rate, which values money later a little below money now. The formula is built up from scratch in Foundations: lifetime value.

Worked example: the discount cohort that grew orders and shrank value

In January Corrib Coffee gains 400 new customers, mostly from organic search. In March it runs a 30% off first order promotion and gains 600. Total orders in spring are up, and as far as Sam can tell, things are going grand.

The cohort view, as the share of each cohort still buying in each month after the first order:

The horizontal axis is months since each customer's first order, the vertical axis is the share of the cohort still buying. Both lines start at 100% because everyone bought once. The takeaway is the gap: the March customers came for the discount and mostly did not stay.

Now the money. Assume a margin of €60 per active customer per year and a discount rate of 10%. Suppose the cohort data, followed for a full year, gives an annual retention of 60% for January and 40% for March.

For January, $CLV = 60 \times \frac{0.6}{1 + 0.1 - 0.6} = 60 \times \frac{0.6}{0.5} = €72$.

For March, $CLV = 60 \times \frac{0.4}{1 + 0.1 - 0.4} = 60 \times \frac{0.4}{0.7} = €34.29$.

If acquiring a customer costs Corrib about €40 in either month, the January customers pay that back with €32 to spare, and the March customers never do. More customers, less value. The monthly total said the opposite.

Two traps to avoid here. Feed margin into $M$, never revenue, or every lifetime value is inflated by the cost of the goods. And never judge a cohort before it has had time to show its shape. How to build the cohort table from your own data is in growth through retention and referral.

An experiment is a decision you write down before you look

Testing was always part of growth hacking. What changes with measurement is the discipline around the test.

The humbling fact first. A team at Microsoft that ran thousands of controlled experiments reported that "only one third of ideas tested at Microsoft improved the metric(s) they were designed to improve", and that for less well understood areas "the statistics are much worse".⁷ Most ideas, including good ones from clever people, do nothing measurable. A test is how you find out which third yours is in, without paying for the other two.

Discipline means four things decided before the test starts: the one metric (in margin, not clicks), the smallest effect worth having, the sample size that can detect it, and the rule for stopping. The sample size for comparing two conversion rates is:

$$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2}$$

Here $n$ is the number of visitors needed in each version, $z_{\alpha/2}$ is 1.96 for a 5% significance level (the chance of seeing an effect that is not there), $z_{\beta}$ is 0.84 for 80% power (the chance of catching an effect that is there), $p_1$ is the current conversion rate and $p_2$ is the rate you hope to reach. The formula and its parts are explained in Foundations: sample size.

The loop runs from top to bottom. The decision box is the only place the result is read, and only once the planned sample is in. Both branches end in the same place, a written note, because a failed test that is recorded is worth more than a winning one nobody remembers.

Worked example: how long Sam's product page test must run

Sam wants to test a new product page. Corrib converts 2.5% of visits to the product pages, and a lift to 2.8% would be worth having. So $p_1 = 0.025$ and $p_2 = 0.028$.

Step one, the brackets:

$$p_1(1 - p_1) + p_2(1 - p_2) = 0.025 \times 0.975 + 0.028 \times 0.972 = 0.024375 + 0.027216 = 0.051591$$

Step two, the top line: $(1.96 + 0.84)^2 = 2.80^2 = 7.84$, and $7.84 \times 0.051591 = 0.40447$.

Step three, the bottom line: $(0.028 - 0.025)^2 = 0.003^2 = 0.000009$.

Step four, divide: $n = \frac{0.40447}{0.000009} = 44{,}941$, so about 45,000 visitors per version.

The product pages get 12,000 visits a week, split evenly, so 6,000 per version. The test needs $\frac{45{,}000}{6{,}000} = 7.5$ weeks, rounded up to eight. Not the two weeks Sam had pencilled in, and not "until it looks good". If eight weeks is too long, the honest options are to test a bolder change (a bigger effect needs fewer visitors) or to accept that this question cannot be answered with Corrib's traffic.

What goes wrong when you check the result every morning and stop on the first good day is set out, with the numbers, in statistical significance and error margins. Why the average order value you test against may itself be misleading is in why averages mislead you.

Automation pays off once the measurement underneath it is right

The original version of this article ended with automation, and that part of the story stands. Once I had the infrastructure to measure success, I started automating campaigns. I integrated customer data from Rails and Solidus with marketing tools such as Mailchimp to trigger campaigns from what customers did, such as abandoned carts, first purchases or a return after a quiet spell. I used Python and SQL to pull transaction data, segment customers by lifetime value and find the campaigns that converted. And I used machine learning to adjust Google Shopping bids from sales performance, margins and competition.

What I would no longer show is the code I published with it. The bidding sketch trained a model to predict sales and then divided the prediction by 100 to get a bid. That number has no meaning as a bid. A sound bid starts from what a click is worth to the business, and that is a short calculation.

$$V = p \times A \times m$$

Here $V$ is the value of one click in euro, $p$ is the conversion rate of clicks into orders, $A$ is the average order value and $m$ is the margin rate. It is an expected value, explained in Foundations: expected value.

Worked example: the most Sam should pay for a click on beans

Clicks on Corrib's bean ads convert at 2.5%, the average bean order is €32 and the margin is 55%, so $V = 0.025 \times 32 \times 0.55 = 0.80 \times 0.55 = €0.44$.

Every click is worth about 44 cent in margin. Paying more than that loses money on average, before any profit at all. A model's job is to estimate $p$ better for each product, audience and hour, not to invent a bid from thin air. And the estimate is only as good as the incrementality behind it: if many of those clicks would have bought anyway, the true value per click is lower still.

How a click auction sets your price is in auction theory and bidding strategy. How the joined customer data underneath all this is built is in data warehousing for marketing automation, and triggering campaigns from behaviour before the purchase is in predictive automation using behavioural segments.

I have also cut two claims from the original. It said the automation adjusted everything "in real time" and "without human intervention". Neither is a goal. A person still decides what the metric is, checks the test and owns the result. And it said this approach outpaces traditional marketing teams, which is a comparison I cannot back with a number, so it goes.

Tactics still matter, but measurement decides which ones stay

Nothing here says tactics are bad. A pop up, a referral offer, a clever integration like Airbnb's: all of them can work. What changed for me is the order of things. The number comes first, the tactic second, and the test decides.

Back to Sam's pop up from the start. With what this article covers he can now judge it properly. He checks the margin on discounted orders, not the revenue. He holds the pop up back from a random share of visitors to see how many of the new sign ups were incremental. He follows the sign ups as a cohort to see whether they come back at full price. And he decides in advance how many visitors the test needs before he reads it. If it survives all four, it stays. That is still growth hacking, in the sense Sean Ellis first meant it. It just keeps its receipts.

If you are starting from scratch, what data science actually delivers for growth is the wider frame for all of this: question, data, model, decision, and a check on whether the decision got better.

Footnotes

1. Sean Ellis, Find a Growth Hacker for Your Startup, Startup Marketing blog, 26 July 2010, https://www.startup-marketing.com/where-are-all-the-growth-hackers/

2. Andrew Chen, Growth Hacker is the new VP Marketing, andrewchen.com, 2012, https://andrewchen.com/how-to-be-a-growth-hacker-an-airbnbcraigslist-case-study/

3. Sean Ellis and Morgan Brown, Hacking Growth, Virgin Digital (Penguin Random House), 2017, publisher's description, https://www.penguin.com.au/books/hacking-growth-9780753545386

4. Google, About Conversion Lift, Google Ads Help, read November 2026, https://support.google.com/google-ads/answer/12003020?hl=en-GB

5. Tom Blake, Chris Nosko and Steven Tadelis, Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment, NBER Working Paper 20171, 2014, published in Econometrica, volume 83, issue 1, 2015, https://www.nber.org/papers/w20171

6. Google, Cohort exploration, Analytics Help, read November 2026, https://support.google.com/analytics/answer/9670133?hl=en-GB

7. Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, KDD 2012, Microsoft, 2012, https://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf