What is a growth hacker, and how to tell real growth work from rebranded marketing

A growth hacker is judged on one number, and that number is growth

The term has been stretched so far that it now covers almost anyone who runs ads. So it helps to go back to the person who coined it. In July 2010 Sean Ellis wrote: "A growth hacker is a person whose true north is growth." Everything such a person does, he went on, "is scrutinized by its potential impact on scalable growth."¹ He also described the method: rank ideas, test them, keep the ones that work, cut the rest, and repeat that process as fast as possible.¹

Two years later Andrew Chen called the role "a hybrid of marketer and coder", someone who answers the question "How do I get customers for my product?" with "A/B tests, landing pages, viral factor, email deliverability, and Open Graph" rather than with a bigger media budget.² In 2017 Ellis and Morgan Brown wrote the method down in the book Hacking Growth, which its publisher describes as an approach that "involves cross functional teams and continuous testing and iteration."³

Notice what all three have in common. None of them defines the role by a channel. Not ads, not email, not social. The role is defined by a target (growth), a method (testing), and a reach that crosses the line between marketing and the product itself. I tell the longer story of where the term came from, and how I moved on from it, in why I moved from growth hacking to data driven eCommerce growth. This article is about the practical part: what the work actually involves, and how you can tell the real thing from a marketing retainer with a new label.

All the examples use one invented business. Sunday's Well Kiln is an online shop in Cork selling handmade ceramics, run by an equally invented owner called Ciara, at sundayswellkiln.example. Every number is made up so I can show every step. Nobody real.

Worked example: two job adverts with the same title

Ciara wants help with growth and writes two drafts of a job advert, both titled "Growth hacker".

The first draft asks for someone to "run our Meta and Google campaigns, grow our Instagram following and send the weekly newsletter". Every task is a channel. Success would be measured in followers, clicks and open rates.

The second draft asks for someone to "raise monthly contribution from new customers, find out why so many people abandon at delivery options, and test changes anywhere from the ads to the checkout". The target is one business number. The method is finding the bottleneck and testing. The reach includes the shop itself.

Only the second is the role Ellis described. The first is a perfectly sound job, a channel marketer. It just is not growth hacking, and paying for the label does not change the work.

The work is a loop, not a channel

If growth hacking is defined by a method, the method is worth seeing in full. It is a loop. You look at the data, find the step where most customers drop away, form a guess about why, and write that guess down with the number it should move. Then you build the smallest change that tests the guess, run it until you have enough data to judge it, and keep it or drop it. Whatever happens, you write down what you learned, and the next turn of the loop starts from that.

The diagram reads from top to bottom. The first five boxes happen before anyone sees a result, which is the point: the question, the metric and the amount of data are fixed in advance. The decision box is the only place the result is read. Both branches end in the log, because a dropped idea that is written down stops the next person from paying to test it again.

Two things make this a loop rather than a list of tactics. First, the step that gets attention is chosen by the data, not by what the team happens to be good at. If the leak is in the checkout, more ad spend only pours more people into the leak. Second, the loop has a memory. The log is how a business gets better at guessing.

Worked example: one turn of the loop at Sunday's Well Kiln

Ciara's shop data for October shows 18,000 product page visits, 1,100 visits that reach the checkout and 400 orders. So 1,100 out of 18,000, about 6.1% of product page visitors, start the checkout, and 400 out of 1,100, about 36.4%, finish it.

Step one, the biggest drop. Most of the loss is between the product page and the checkout. Ciara reads 50 recent customer emails and finds a good few asking the same thing: "Will it arrive before the weekend?"

Step two, the hypothesis: "Showing an estimated delivery date on every product page will raise the share of product page visitors who order, from 2.2% to 2.6%." The current rate comes from 400 orders out of 18,000 visits, which is about 2.2%.

Step three, the ranking. It is one of four ideas on her list, and the next section shows how to rank them.

Step four, the plan. The metric is orders per product page visit. The smallest effect worth having is the jump to 2.6%. The sample size needed follows from the formula in Foundations: sample size, and she writes it down before the test starts.

Step five, the build. The delivery estimate needs the dispatch rules from the warehouse and a line of code on the product template. That is product and engineering work, not marketing, which is the subject of a later section.

Steps six and seven, run and decide. If the lift holds at the planned sample, it ships. If not, it goes in the log as "delivery date on product page: no measurable effect at this traffic", and that is still worth knowing.

Measurement comes before the first experiment

A loop that runs on wrong numbers just gets you to the wrong answer faster. So the first job of anyone doing real growth work is not an experiment at all. It is checking that the numbers can be trusted.

That means three things. Every order counted once, in the place that actually takes the money (the shop's own database), not only in an analytics tool. Every important step tracked with an event that fires reliably, with consent handled properly. And the source of each visit recorded with consistent tags, so that a change in the mix of traffic does not look like a change in behaviour. The detail is in event tracking and consent mode and mastering UTM tracking.

The old version of this article put it in three steps: do not guess, instrument, then act. I still agree with the order.

Worked example: the 110 orders that analytics could not see

Ciara's analytics tool reports 410 orders for October. Her shop database records 520 paid orders in the same month.

Step one, the gap: $520 - 410 = 110$ orders, so $\frac{110}{520} = 0.212$, about 21% of real orders are invisible to the analytics tool. The percentage arithmetic is in Foundations: percentages.

Step two, the causes. She finds two. Visitors who decline cookies are, correctly, not tracked by the analytics tool. And customers who pay through a payment provider's own page sometimes do not come back to the thank you page, so the purchase event never fires.

Step three, the consequence. Any experiment judged in the analytics tool would be judged on four fifths of the orders, and not a random four fifths. If the new product page happens to send more people through the payment provider that loses the return trip, the test would show a drop that is not real.

The fix is to count the outcome from the shop database and use analytics only for the steps before the order. Until that is done, every test result is a guess with a decimal point. If you suspect your own numbers, is your dashboard lying to you walks through a ten minute check.

Most ideas fail, so the value is in the programme

Here is the fact that separates real growth work from confident marketing. Most ideas do not work. A team at Microsoft that ran a very large number of controlled experiments reported that "only one third of ideas tested at Microsoft improved the metric(s) they were designed to improve", and that for areas that are not well understood "the statistics are much worse".⁴ The same paper quotes Jim Manzi's figure that Google ran about 12,000 randomised experiments in 2009, with only about 10% of them leading to business changes.⁴

These are some of the most data rich companies on earth. Your ideas, and mine, are not going to beat them by much. So a single experiment is a bet that usually loses. The value comes from running many of them, at a cost per experiment low enough that the winners pay for the losers. That is an expected value calculation.

$$E[V] = N \times (p \times W - c)$$

Here $E[V]$ is the expected value of a year of experiments in euro of margin, $N$ is the number of experiments run in the year, $p$ is the share of experiments that win, $W$ is the margin a winning change adds over a year, and $c$ is the cost of designing, building and running one experiment. The bracket is the expected value of one experiment: the chance of winning times what a win is worth, minus what every experiment costs whether it wins or not. The idea is built up from scratch in Foundations: expected value.

Worked example: is a year of experiments worth it for Sunday's Well Kiln

Assume, for this example, that Sunday's Well Kiln turns over €720,000 a year at a margin rate of 40%, and that a typical winning change lifts conversion by 3% of its current level across the shop. Those are the example's assumptions, chosen to be modest.

Step one, what a win is worth. A 3% lift on €720,000 is $720{,}000 \times 0.03 = €21{,}600$ of extra revenue, and at a 40% margin that is $21{,}600 \times 0.40 = €8{,}640$ of margin a year. So $W = 8{,}640$. I count only the first year, which is conservative.

Step two, what an experiment costs. Assume €1,000 of time and tooling per experiment, so $c = 1{,}000$.

Step three, one experiment at three different win rates, taken from the range in the Microsoft paper.

At a 10% win rate: $0.10 \times 8{,}640 - 1{,}000 = 864 - 1{,}000 = -€136$. Each experiment loses money on average.

At a 20% win rate: $0.20 \times 8{,}640 - 1{,}000 = 1{,}728 - 1{,}000 = €728$.

At one in three: $\frac{1}{3} \times 8{,}640 - 1{,}000 = 2{,}880 - 1{,}000 = €1{,}880$.

Step four, the programme. Take the middle case, 20%, and 24 experiments in a year, two a month: $E[V] = 24 \times 728 = €17{,}472$ of expected margin.

Two lessons sit in those numbers. If your ideas are poor, or your experiments are expensive, the programme loses money however hard everyone works. And the expected value depends heavily on two things that have nothing to do with ad copy: how little an experiment costs to build, and how many can be run in a year. Both depend on access, which is the next section.

The expected value also says nothing about variance. In a year of 24 experiments at a 20% win rate, you would expect about five winners, but a run of 15 losers in a row is entirely possible. If a bad quarter would sink the business, the programme needs to be smaller, not braver. That trap is explained in decision theory for founders.

Without product and engineering access, there is no loop

Look back at the delivery date idea. The ad account could not test it. Neither could the email tool. It needed the product page template, the dispatch rules and someone able to change both safely. The same is true of most ideas with a real chance of moving the number: a shorter checkout, a clearer delivery promise, a better search on the shop, a referral offer built into the order confirmation, a fix for a form that fails on one browser.

This is why Chen's "hybrid of marketer and coder" matters more than it sounds.² Someone who can only touch the ads can only test ads. Someone who can read the shop's data, change the shop's pages and ship the change through a proper release can test the whole journey. The old version of this article listed the areas: the product (onboarding, referral loops, retention triggers), engineering (APIs, webhooks, automation), marketing (campaigns, targeting, testing cadence), data (tracking, attribution, cohorts) and behavioural psychology (friction, habit, reward). Few people are strong in all five. In my view, as I wrote then, it takes years of real work across them, not a course, and that is what makes the role rare.

It also explains speed. Every handover is a delay. If one test needs a ticket to a developer, a slot in a designer's week and a meeting to agree the copy, it takes weeks to start, and the opportunity may have moved by then. I have trimmed one claim from the old version, that a growth hacker does all of this "in hours, not weeks". Some changes do take hours. A careful change to a checkout, tested and released safely, does not, and I would not want it to.

Worked example: the same ideas with and without access

Ciara compares two ways of running the programme from the previous section, at the same 20% win rate and the same €8,640 for a win.

With access. The person running the loop can change the shop directly, through a tested release. Assume each experiment costs €1,000 and two can be run a month, so $N = 24$ and $c = 1{,}000$. The expected value is $24 \times (0.20 \times 8{,}640 - 1{,}000) = 24 \times 728 = €17{,}472$.

Without access. Every change goes into a queue at a separate development agency, with its own quote and its own release date. Assume each experiment then costs €1,600 once the agency's time is added, and only eight fit into the year. So $N = 8$ and $c = 1{,}600$. The expected value is $8 \times (0.20 \times 8{,}640 - 1{,}600) = 8 \times (1{,}728 - 1{,}600) = 8 \times 128 = €1{,}024$.

The vertical axis is the expected margin from a year of experiments, in euro. Each bar is one way of running the same programme, with the same ideas and the same chance of a win. The takeaway: the difference is not talent or creativity. It is the number of experiments and their cost, and both are set by who can change the product. Nearly the same ideas are worth seventeen times as much when they can be tested quickly.

Seeing the system, not the symptom

The second skill is diagnosis. A campaign that stops working is a symptom. The cause can sit anywhere: in the targeting, in the tracking, in the page, in the checkout, in the margin. The old version of this article put the questions well, and I still ask them. Was the wrong intent targeted? Is the data being tracked correctly? Is the form failing on one browser? Is the call to action out of sight on small screens? Is the lifetime value assumption making the ad spend look better than it is?

A channel marketer, looking only at the ad account, can only change what is in the ad account: the headline, the image, the bid. That is not a criticism of the person. It is a limit of their view. The growth version of the job is to follow the number down through every layer until it stops making sense, and fix it there. The framework I use for finding the layer is in the pirate metrics framework, and how features and funnels feed each other is in where product ends and growth begins.

Worked example: the headline that was never the problem

Ciara's Meta campaign sends 10,000 clicks a month to the shop. Its conversion rate falls from 2.4% to 1.6% in September. The obvious move is a new headline.

Step one, split by device and browser. Half the clicks come from Safari on iPhones, half from everything else.

Before September, Safari converted at 2.2% and everything else at 2.6%. That gives $5{,}000 \times 0.022 + 5{,}000 \times 0.026 = 110 + 130 = 240$ orders, which is 2.4% of 10,000.

In September, everything else still converts at 2.6%, 130 orders. Safari has dropped to 0.6%, $5{,}000 \times 0.006 = 30$ orders. Total 160, which is 1.6%.

Step two, find the cause. The drop starts the day a theme update went live. On Safari, the address lookup in the checkout no longer fills the form, and customers give up.

Step three, price it. The campaign lost $240 - 160 = 80$ orders a month. At an average order of €45 and a 40% margin, that is $80 \times 45 \times 0.40 = €1{,}440$ of margin a month.

A new headline would have changed nothing at all, and the test of it would have looked like a failure of the ad. The fix was one line in the checkout. Finding it needed access to the analytics by browser, the release history and the checkout code. That is the system, not the symptom.

How to tell real growth work from rebranded marketing

When I first wrote this article, I said that most agencies claiming the growth hacking label are still digital marketing shops with louder branding. That is my view, from what I have seen, not a statistic, and I do not name anyone. You do not need to trust my view either. The difference shows up in the answers to a few questions, and in whether the evidence behind those answers exists.

Here are the questions I would ask, and what a good answer contains.

1. What number will you be judged on? A good answer is one business number, ideally in margin, agreed in writing. A weak answer is a list of channel metrics: reach, clicks, followers.
2. Can I see your experiment log? A good answer is a real log, anonymised if needed, with the hypothesis, the planned sample, the result and the decision for each test. It should contain far more losers than winners, because that is what honest testing looks like.⁴
3. How do you decide when a test is finished? A good answer is a sample size fixed before the start. A weak answer is "when it reaches significance". Checking every day and stopping at the first good result inflates false positives badly. In one worked example, Evan Miller shows a change that does nothing being declared a winner 26.1% of the time, more than five times the 5% the tester thought they were accepting.⁵
4. What access do you need? A good answer includes the shop's data and the ability to change pages through a safe release, or a clear plan with whoever holds them. A weak answer is access to the ad accounts only, which limits every test to the ads.
5. Tell me about a test that failed and what you learned. A good answer is specific, with numbers. A weak answer is a story about a campaign that went well.

None of these needs a badge or a certificate, and a badge does not answer any of them. The wider checklist for choosing outside help, including contracts, account ownership and reporting you can check yourself, is in the guide to picking a growth or digital marketing agency. How the way you pay someone shapes what they optimise for is in principal agent theory in agencies and retainers.

Worked example: Ciara asks two offers the same five questions

Ciara has two proposals, both titled "growth". She asks each the five questions and marks an answer as a pass only when evidence comes with it.

Offer A says it will be judged on "return on ad spend and audience growth". It says it "A/B tests everything", but cannot show a log. It stops tests "once the platform shows a winner". It asks for admin on the ad accounts and nothing else. Its failure story is a campaign that "did not scale as fast as hoped". That is zero passes out of five.

Offer B proposes to be judged on contribution from new customers. It shows a log of 30 past tests from other shops, anonymised, with seven kept and 23 dropped. It explains its sample size calculation and states the stopping rule in the proposal. It asks for read access to the shop database and a release process for small changes to the shop. Its failure story is a free delivery threshold that raised orders but lowered margin, with the numbers. Five passes out of five.

Step one, the win rate in B's log: $\frac{7}{30} = 0.233$, about 23%. That sits inside the range the Microsoft paper describes, which is a sign of honesty, not weakness.⁴

Step two, the decision. Offer A may well run good ads. It is a channel marketing offer, and Ciara can buy it as that, at a channel marketing price. Offer B is the work Ellis described. Ciara does not need to judge anyone's character. The evidence did it for her.

What growth work cannot do for you

Three limits are worth knowing before you hire anyone for this.

It cannot test what your traffic cannot measure. A small shop may need many weeks to detect a modest lift, and some questions simply cannot be answered at low volume. The honest move then is to test bolder changes or to decide on judgement, and say so. The numbers for your own traffic are in how much data do you actually need, and the significance side is in statistical significance and error margins.

It cannot rescue a product people do not want. The loop finds and removes friction. If customers try something once and never come back, the job is the product, not the funnel.

And it does not replace the people who run the business. A loop needs someone to decide what the number is, to accept the losers and to live with the winners. That person is usually you.

Worked example: when Sunday's Well Kiln should not run the test

Ciara's wholesale page gets about 600 visits a month and produces four trade enquiries. She wants to test a new layout.

Step one, the rate: $\frac{4}{600} = 0.0067$, about 0.67%.

Step two, the volume. Even a large lift, say to 1.0%, would mean about six enquiries a month instead of four. A difference of two enquiries a month is well inside the normal wobble of such small counts, as Foundations: sampling explains, and the sample size formula would ask for far more visits than the page gets in a year.

Step three, the decision. She does not run a test. She phones five trade customers, asks what nearly stopped them from enquiring, changes the page on that evidence and watches the enquiries over the next quarter. That is still growth work. It is just honest about what the data can and cannot tell her.

The short version

A growth hacker, in the original sense, is judged on growth, works in a loop of measured experiments, and can reach beyond the ad accounts into the product and the data. The skills are diagnosis across layers, measurement you can trust, and the access and speed to test many ideas at a low cost each. Most ideas fail, so the value is in the programme, and a programme only pays when experiments cost little and run often.

To tell the real thing from rebranded marketing, ask for the number, the log, the stopping rule, the access and a failure. The answers, and the evidence behind them, tell you more than any title on a proposal.

Footnotes

1. Sean Ellis, Find a Growth Hacker for Your Startup, Startup Marketing blog, 26 July 2010, https://www.startup-marketing.com/where-are-all-the-growth-hackers/

2. Andrew Chen, Growth Hacker is the new VP Marketing, andrewchen.com, 2012, https://andrewchen.com/how-to-be-a-growth-hacker-an-airbnbcraigslist-case-study/

3. Sean Ellis and Morgan Brown, Hacking Growth, Virgin Digital (Penguin Random House), 2017, publisher's description, https://www.penguin.com.au/books/hacking-growth-9780753545386

4. Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, KDD 2012, Microsoft, 2012, https://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf

5. Evan Miller, How Not To Run an A/B Test, evanmiller.org, 18 April 2010, https://www.evanmiller.org/how-not-to-run-an-ab-test.html