Unit, integration and end to end tests: what to test where

TL;DR

Difficulty: Intermediate

Test every behaviour at the lowest level that can prove it.

  • Unit tests check one rule alone, in milliseconds, so you write many.
  • Integration tests check the parts working together, with fakes for services you do not own.
  • End to end tests drive a real browser through the journeys that earn money, and stay few because they are slow and fragile.
  • Give each level a speed budget.
  • Treat flaky tests as bugs: with n tests each flaky with chance p, a run is red for no reason with chance 1 minus (1 minus p) to the power n, about 63% for 200 tests at 0.5%.

Once you write tests first, the next question is which tests, and how many of each. Here is a plain answer: the three levels, the pyramid and its critics, the stand ins for outside services, a speed budget per level, and why a few flaky tests can make a whole suite worthless. All of it shown on one tea shop checkout, tested at three levels.

Which tests do I need, and how many of each?

Once you have written your first test before the code, a second question arrives almost at once. There are many kinds of test, and every guide seems to name them differently. Unit, integration, functional, acceptance, system, end to end. Which ones does a real product need? How many of each? And where does a particular check belong?

This article answers that, plainly, for developers and for the business owners who pay for the work. If you have not met the idea of an automated test yet, start with the first chapter of this series, test driven development explained. Here I assume you know what a test is (arrange, act, assert) and want to know where to put the next one.

The example business all the way through is Fernhill Tea, an invented online tea shop in Bristol at fernhilltea.example. It sells loose tea and monthly subscriptions, takes card payments through a payment provider, accepts discount codes, keeps track of stock and sends order emails. Its developer is Priya. Every number about Fernhill below is made up for the example.

Three levels, defined plainly

Tests differ mainly in one thing: how much of the real system they switch on at once. The more they switch on, the more they prove, and the slower and more fragile they become. Three levels cover almost everything.

Unit tests

A unit test checks one small piece of logic on its own: a function, a method, a class. It runs in the same process as the code, with no database, no network and no browser. It is fast (a few milliseconds) and when it fails it points at one place.

Google's engineering book calls these small tests and defines them by what they may touch: small tests run in a single process1. That definition is more useful than the name. If a test needs a database, it is no longer small, whatever you call it.

Integration tests

An integration test checks that several parts work together: your code and the database, your code and the mailer, a controller and the model behind it. It still runs on one machine, but now real pieces talk to each other. Google's medium tests may span several processes and make network calls, but only to the same machine1. In Rails, integration tests live in test/integration and drive the app through its routes, while model and service tests use the same base class as unit tests but usually touch the real test database2.

What an integration test does not do is call services you do not own, such as the real payment provider. Those are replaced by a stand in, which I come to below.

End to end tests

An end to end test drives the whole application the way a customer does: a real browser opens a page, fills in fields, clicks buttons and reads the result. Rails calls these system tests. They live in test/system, use the Capybara library to steer the browser, and by default bin/rails test does not even run them: you ask for them with bin/rails test:system or bin/rails test:all2. In JavaScript projects the same job is often done by Playwright.

End to end tests prove the most, because nothing is faked between the customer and the database. They are also the slowest, often several seconds each, and the most likely to fail for reasons that have nothing to do with your code.

Example: one discount code, three levels

Fernhill Tea runs a code, LEAF10, for 10% off. Here is what each level would check.

  • Unit: given a basket of £44.00 and the code LEAF10, the price calculator returns £39.60. Pure arithmetic, no database. It runs in about a millisecond.
  • Integration: when an order is placed with LEAF10, the saved order in the database records the code, the discount of £4.40 and the right total, and the code's usage counter goes up by one. Real database, fake payment provider.
  • End to end: a customer types LEAF10 into the checkout in a real browser, sees "LEAF10 applied, you save £4.40" and pays the reduced amount.

All three are about the same rule. Each proves something the others cannot. The unit test proves the sum. The integration test proves the sum is stored. The end to end test proves a person can actually use it. The question is how many of each you need, and that is where the pyramid comes in.

The test pyramid, and why it has that shape

The best known answer is the test pyramid. Mike Cohn described it in conversation with Lisa Crispin around 2003 and 2004 and made it famous in his 2009 book Succeeding with Agile, as the "test automation pyramid"3. The picture is simple: a wide base of unit tests, a middle layer of service or integration tests, and a narrow tip of tests through the user interface.

Ham Vocke's long article on Martin Fowler's site boils it down to two rules: write tests with different granularity, and the more high level you get, the fewer tests you should have4. Fowler gives the reason for the narrow top: tests through the user interface are "brittle, expensive to write, and time consuming to run"3.

Google's book puts rough numbers on it: around 80% unit tests, 15% integration tests and 5% end to end tests, with the explicit warning that every team's mix will differ1. It also names the shape to avoid, the ice cream cone: many end to end tests and few unit or integration tests underneath. Such suites, it says, "tend to be slow, unreliable, and difficult to work with"1.

Example: counting the tea shop's suite

Priya counts the tests in Fernhill Tea's repository. These numbers are invented for the example.

A bar chart of an invented example suite. 640 unit tests, 140 integration tests and 18 end to end tests. The bars shrink from bottom level to top level, the shape of the test pyramid. Tap or click to open full screen and move around

That is 798 tests in all. The shares are 640 out of 798, or about 80%, then 140 out of 798, about 18%, and 18 out of 798, about 2%. It is close to the shape Google suggests, a little heavier in the middle and a little lighter at the top. That is fine. The pyramid is a rule of thumb, not a quota. What matters is that the base is wide and the tip is narrow, and the next section on speed shows why.

The critics: the testing trophy

Not everyone draws a pyramid. Kent C. Dodds, who teaches JavaScript testing, proposed the testing trophy in 20185. From the bottom up it has four layers: static checks (type checking and linting, which catch typos and wrong types before any test runs), then unit tests, then a large integration layer, then a small end to end cap. Its slogan, borrowed from Guillermo Rauch, is "Write tests. Not too many. Mostly integration."5

The argument is fair. In a modern front end, a unit test of one tiny component often proves very little, because the bugs live in how components, state and the network fit together. A test that renders a whole form and fills it in, without a real browser, catches far more for its cost.

So who is right? Both, for their own ground. The pyramid grew out of server side code, where business rules sit in plain objects that are easy to test alone. The trophy grew out of browser code, where the interesting behaviour is in the wiring. Both agree on the parts that matter most: many fast tests at the bottom, few slow browser tests at the top, and a deliberate choice about the middle.

Example: two shapes in one shop

Fernhill Tea's back end is Rails. Its pricing, stock and subscription rules sit in small Ruby classes, so Priya tests them with many unit tests: pyramid shaped. The subscription page, built as a React component on a Next.js front end, lets a customer pick a tea, a bag size and a delivery rhythm. Priya tests it mostly by rendering the whole form, choosing options and checking the summary it shows, with the network call to the back end replaced. That part of the suite is trophy shaped. Neither shape is wrong. Each fits where the bugs actually live.

Test doubles: stubs, mocks, fakes and spies

Integration tests replace the services you do not own. The general name for any stand in is a test double, after a stunt double in a film. Martin Fowler's short glossary, based on Gerard Meszaros's vocabulary, gives five kinds of double6.

  • Dummy: an object passed around to fill a parameter list, never actually used.
  • Stub: gives canned answers to the calls made during the test, and ignores everything else.
  • Spy: a stub that also records how it was called, so the test can check afterwards.
  • Mock: set up in advance with the calls it expects, and fails the test if the calls differ.
  • Fake: a real, working implementation with a shortcut, such as an in memory database, not fit for production.

There is one more distinction worth knowing, from Fowler's essay Mocks Aren't Stubs. A test can check the state at the end (is the order saved, is the stock lower?) or it can check the behaviour on the way (did the code call charge with these arguments?). Mocks always check behaviour7. State checks tend to survive refactoring better, because they do not care how the code reached the result. Behaviour checks are right when the call itself is the point, such as "we must send exactly one charge to the payment provider".

Example: the payment provider, four ways

Fernhill Tea's checkout calls a payment gateway object with charge(amount_pence:, token:, idempotency_key:). Priya can replace it in different ways.

A stub only answers:

stub_gateway = Object.new
def stub_gateway.charge(**) = { ok: true, reference: "stub_1" }

The first line makes an empty object. The second gives it one method, charge, which accepts any keyword arguments and always says the payment worked. Useful when the test is about something else, for example the confirmation page.

A mock knows what it expects and complains otherwise:

gateway = Minitest::Mock.new
gateway.expect(:charge, { ok: true, reference: "m_1" },
               amount_pence: 2895, token: "tok_visa", idempotency_key: String)
PlaceOrder.new(gateway:).call(basket:, card_token: "tok_visa")
gateway.verify

expect says: one call to charge is coming, with exactly 2,895 pence, this token and any string as the key, and here is what to return. verify at the end fails the test if that call never happened. Right for "we charge exactly once, the right amount".

A fake, which is also a spy, behaves like a small real provider and keeps a record:

# test/support/fake_payment_gateway.rb
class FakePaymentGateway
  attr_reader :charges

  def initialize(decline: false)
    @decline = decline
    @charges = []
  end

  def charge(amount_pence:, token:, idempotency_key:)
    @charges << { amount_pence:, token:, idempotency_key: }
    return { ok: false, reason: "card_declined" } if @decline

    { ok: true, reference: "fake_#{@charges.size}" }
  end
end

It starts with an empty list of charges. Every call adds an entry, so a test can later read charges and check the amount (that is the spy part). Built with decline: true, it refuses every card the way a real provider would. Priya uses this fake in most of her integration tests, because one small class serves the happy case and the declined case alike.

The danger with every double is drift: the real provider changes and the double does not. The article on testing APIs between applications shows how contract tests keep a double honest. Google's book makes the same point with numbers: two doubles that are each slightly wrong multiply their errors once the parts are combined8.

Speed budgets: what each level may cost

A test you do not run protects nothing. Google's book puts it bluntly: the slower a test is, the less often an engineer runs it8. Vocke says a decent machine runs thousands of unit tests within a few minutes, and that a pipeline should run the fast tests first, so a mistake shows up as early as possible4.

A speed budget turns that into a rule you can check. You decide, per level, how long one test may take and how long the whole level may take, and you treat a breach like a failing test. There is no official standard for the numbers. Here is a sensible starting point, which you adjust to your project:

LevelOne testThe whole levelWhen it runs
Unitunder 10 millisecondsunder 30 secondson every save
Integrationunder 1 secondunder 2 minutesbefore every commit
End to endunder 20 secondsunder 10 minuteson every push, in continuous integration

For comparison, Google's large tests get a default timeout of 15 minutes or an hour8. A small shop does not need anything near that, and should not want it.

Example: timing the tea shop's suite

Priya times her suite on her laptop. The figures are invented for the example.

  • 640 unit tests at about 10 milliseconds each: 640 × 0.01 = 6.4 seconds.
  • 140 integration tests at about 0.25 seconds each: 140 × 0.25 = 35 seconds.
  • 18 end to end tests at about 12 seconds each: 18 × 12 = 216 seconds.

The whole suite takes 6.4 + 35 + 216 = 257.4 seconds, about four and a quarter minutes. Look at where the time goes: the 18 browser tests are about 2% of the tests and 216 out of 257.4, about 84%, of the time. Every new end to end test adds roughly as much time as a thousand unit tests. That is the pyramid's argument in one line of arithmetic, and it is why Priya asks "could a lower level prove this?" before she writes another browser test.

Worked example: Fernhill Tea's checkout at three levels

Now the full example, every step shown. The behaviour: a customer buys two 100 gram bags of Assam Breakfast at £12.50 each. Below £40 the shop charges £3.95 shipping. So the customer pays 2 × £12.50 + £3.95 = £28.95. Money is stored as whole pence to avoid rounding errors, so that is 2,895 pence.

Level one: a unit test of the price calculation

The first test checks the arithmetic and nothing else.

# test/services/price_calculator_test.rb
require "test_helper"

class PriceCalculatorTest < ActiveSupport::TestCase
  test "adds 395 pence shipping below 4000 pence" do
    calc = PriceCalculator.new(lines: [{ price_pence: 1250, quantity: 2 }])
    assert_equal 2895, calc.total_pence
  end

  test "ships free from 4000 pence" do
    calc = PriceCalculator.new(lines: [{ price_pence: 2000, quantity: 2 }])
    assert_equal 4000, calc.total_pence
  end

  test "applies the discount before deciding on shipping" do
    calc = PriceCalculator.new(lines: [{ price_pence: 2200, quantity: 2 }], discount_percent: 10)
    assert_equal 3960 + 395, calc.total_pence
  end
end

Line by line: require "test_helper" loads the test setup. The class groups the tests for one subject. Each test "..." do block is one behaviour, named in plain words. Inside, the first line arranges a calculator with a basket, and assert_equal checks the expected number against the actual one. The third test pins down a business decision: £44.00 minus 10% is £39.60, which is below £40, so shipping is charged. Someone had to decide that, and the test now records the decision.

The code that makes these pass is small:

# app/services/price_calculator.rb
class PriceCalculator
  SHIPPING_PENCE = 395
  FREE_SHIPPING_FROM_PENCE = 4000

  def initialize(lines:, discount_percent: 0)
    @lines = lines
    @discount_percent = discount_percent
  end

  def total_pence
    subtotal = @lines.sum { |line| line[:price_pence] * line[:quantity] }
    discounted = subtotal - (subtotal * @discount_percent / 100)
    discounted + (discounted >= FREE_SHIPPING_FROM_PENCE ? 0 : SHIPPING_PENCE)
  end
end

It adds up price times quantity for every line, takes the discount off, then adds shipping only when the discounted amount is below £40. These three tests run in a few milliseconds together. Priya can run them on every save.

Level two: an integration test of the order service

The second test checks that placing an order really works with the database and the mailer, while the payment provider is the fake from earlier.

# test/services/place_order_test.rb
require "test_helper"

class PlaceOrderTest < ActiveSupport::TestCase
  include ActionMailer::TestHelper

  setup do
    @tea = products(:assam_breakfast) # fixture: 1250 pence, 10 in stock
    @basket = Basket.create!(lines: [BasketLine.new(product: @tea, quantity: 2)])
  end

  test "a paid order is saved, stock goes down and one email is queued" do
    gateway = FakePaymentGateway.new

    assert_enqueued_emails 1 do
      order = PlaceOrder.new(gateway:).call(basket: @basket, card_token: "tok_visa")
      assert order.paid?
    end

    assert_equal 2895, gateway.charges.first[:amount_pence]
    assert_equal 8, @tea.reload.stock
  end

  test "a declined card leaves the order unpaid and the stock untouched" do
    gateway = FakePaymentGateway.new(decline: true)

    order = PlaceOrder.new(gateway:).call(basket: @basket, card_token: "tok_declined")

    refute order.paid?
    assert_equal 10, @tea.reload.stock
  end
end

Line by line: include ActionMailer::TestHelper brings in the email assertions. setup runs before each test. It loads the Assam tea from the test data (a fixture, a small file of sample records) and saves a real basket with two bags in the test database. The first test builds the fake gateway, then wraps the order in assert_enqueued_emails 1, which fails unless exactly one email was queued for sending. Inside, it places the order and checks it is paid. Afterwards it reads the fake's record to check that exactly 2,895 pence were charged, and reloads the tea from the database to check the stock fell from 10 to 8.

The second test uses a fake that declines. It checks the two things a declined payment must never do: mark the order as paid, or take stock out of the shelf. This is the first glimpse of the unhappy path, which gets its own chapter, testing the unhappy path.

These tests take a fraction of a second each, because the database is local and nothing leaves the machine.

Level three: one end to end test of the checkout

The last test drives a real browser through the whole journey.

# test/system/checkout_test.rb
require "application_system_test_case"

class CheckoutTest < ApplicationSystemTestCase
  test "a customer buys two bags of Assam and sees the confirmation" do
    visit product_path(products(:assam_breakfast))
    fill_in "Quantity", with: 2
    click_on "Add to basket"
    click_on "Checkout"

    fill_in "Email", with: "sam@fernhilltea.example"
    click_on "Pay £28.95"

    assert_text "Thank you, your order is confirmed"
    assert_text "£28.95"
  end
end

Line by line: visit opens the product page in the browser. fill_in types into the field labelled Quantity, and click_on presses the button or link with that text, exactly as a customer would. Then the test fills in an email address on the checkout page and pays. The two assert_text lines check what the customer sees at the end. In Fernhill Tea's test environment the app is configured to use the same fake gateway, so no real card is charged.

Notice what this test does not do. It does not check the discount rules, the stock count or the declined card. Those are proven faster and more precisely below it. The end to end test proves one thing only: a person can get from a product page to a paid order. That is why one is enough here.

Payment providers also publish test card numbers for their sandbox, the provider's own test environment. Stripe's list starts with 4242 4242 4242 4242, for example9. A separate, slower run against that sandbox (say once a night) checks that the real integration still works, without slowing down every push.

If the front end is a separate JavaScript application, the same journey in Playwright reads almost the same:

// e2e/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('a customer buys two bags of Assam', async ({ page }) => {
  await page.goto('/teas/assam-breakfast');
  await page.getByLabel('Quantity').fill('2');
  await page.getByRole('button', { name: 'Add to basket' }).click();
  await page.getByRole('link', { name: 'Checkout' }).click();
  await page.getByLabel('Email').fill('sam@fernhilltea.example');
  await page.getByRole('button', { name: 'Pay £28.95' }).click();
  await expect(page.getByText('Thank you, your order is confirmed')).toBeVisible();
});

getByLabel and getByRole find elements the way a screen reader would, by their label and role, which keeps the test close to what a person experiences. The final expect waits until the confirmation is visible, which matters a great deal in the next section.

This checkout is exactly the kind of journey that must never break. How to find those journeys and guard them is the subject of the next chapter, the critical path in software.

Flaky tests: the tests that cry wolf

A flaky test is one that sometimes passes and sometimes fails on exactly the same code. Google defines a flaky result as a test that shows both a passing and a failing result with the same code10.

Flaky tests do more damage than their number suggests, because they teach people to ignore red. Once a team learns that "it is probably just that test again", a real failure gets waved through too. John Micco reported on Google's testing blog in 2016 that about 1.5% of all their test runs reported a flaky result, that almost 16% of their tests had some level of flakiness, and that about 84% of the transitions from pass to fail they observed involved a flaky test10. In other words, most of the time a test turned red, the code was not the reason. Google's book adds that as you approach 1% flakiness, tests begin to lose their value1.

Where does flakiness come from? The first large study, by Luo and colleagues at the University of Illinois in 2014, examined 201 fixes of flaky tests in 51 open source projects. Of the 161 they could classify, 45% were async wait (the test checked a result before the asynchronous work had finished), 20% were concurrency (several threads interfering with each other) and 12% were test order dependency (a test that only passes if another one ran before it)11.

Browser tests are especially exposed to the first cause, because pages load and change in their own time. That is why both Capybara and Playwright wait for you. Capybara retries finding an element until it appears, up to a configurable maximum wait time12. Playwright checks that an element is visible, stable, able to receive clicks and enabled before acting on it, and its web first assertions retry until the condition is met13. A test that uses those tools as intended avoids most async wait flakes. A test that adds sleep 2 instead invites them.

The maths of a suite with flaky tests

Why does a little flakiness in each test add up to a lot of red builds? Because a build is red if any single test fails. Suppose each of nn tests fails at random, independently of the others, with a small probability pp. The chance that at least one of them fails, so that the whole run is red although nothing is wrong, is:

Pred=1−(1−p)nP_{\text{red}} = 1 - (1 - p)^{n}

Here PredP_{\text{red}} is the probability that a run fails without a real bug, pp is the chance that one test fails at random in one run, and nn is the number of such tests in the suite. Step by step: 1−p1 - p is the chance that one test passes, (1−p)n(1 - p)^{n} is the chance that all nn pass (the probabilities multiply because the tests are independent), and one minus that is the chance that not all of them pass. If the idea of multiplying independent chances is new, the Foundations: probability section explains it from scratch.

Take 200 tests that each flake once in 200 runs, so p=0.005p = 0.005:

1−(1−0.005)200=1−0.367=0.6331 - (1 - 0.005)^{200} = 1 - 0.367 = 0.633

With n=200n = 200 and p=0.005p = 0.005, about 63% of runs fail for no reason. Each test looks reliable on its own, 99.5% of the time, and the suite as a whole is red more often than green.

A second, simpler formula tells you how many flaky failures to expect in one run:

E=n⋅pE = n \cdot p

Here EE is the expected number of tests that fail at random in one run, nn the number of tests and pp the chance of a random failure per test. Expected value is explained in Foundations: expected value. Micco's own example fits it: a project with about 1,000 tests and a flaky rate of 1.5% has E=1000×0.015=15E = 1000 \times 0.015 = 15, so about 15 tests will likely fail in each release run10.

A line chart calculated from the formula one minus one minus p to the power n. For a flaky rate of 0.1% per test the chance of a false red run grows from 1% at 10 tests to 39% at 500 tests. For 0.5% it grows from 5% to 92%. For 1% it grows from 10% to 99%. Tap or click to open full screen and move around

The chart is calculated from the formula, not measured. Read along any line and the lesson is the same: flakiness that looks harmless in one test becomes the normal state of a large suite.

Example: Fernhill Tea's flaky browser tests

Say 2% of Fernhill Tea's end to end runs fail at random, so p=0.02p = 0.02, and there are n=18n = 18 of them. Then Pred=1−0.9818=1−0.695=0.305P_{\text{red}} = 1 - 0.98^{18} = 1 - 0.695 = 0.305. Almost one push in three turns red without a real bug. Priya finds the main cause: one test checks the order confirmation email with sleep 2 and then looks at the page, and on a slow run the page has not updated yet. She replaces the sleep with assert_text "Confirmation sent", which waits until the text appears. After a few more fixes like it, the flaky rate falls to 0.2%, so p=0.002p = 0.002, and the chance of a false red run drops to 1−0.99818=0.0351 - 0.998^{18} = 0.035, about one push in 28.

Quarantine, then fix: what to do with a flaky test

You cannot fix every flake the moment you see it, and you must not let it keep poisoning the build. The usual answer has four steps.

  1. Confirm it is flaky. Run the test again on the same code. Playwright does this for you if retries are switched on, and labels a test that failed first and passed on a retry as "flaky", separate from "passed" and "failed"14.
  2. Quarantine it. Move it out of the gate that blocks a release, for a short time, with a ticket and a name next to it. Google does this automatically when a test's flakiness gets too high: the tool removes it from the critical path and files a bug for developers10.
  3. Fix the cause, not the symptom. Look first at the three big causes from the Illinois study: a missing wait, threads or jobs racing each other, and tests that depend on order or on data left behind by another test11.
  4. Bring it back. Return the test to the gate once it has passed many runs in a row. A quarantined test with no owner and no deadline is a deleted test with extra steps.

Do not simply turn up the automatic retries until the build is green. Retries hide flakes. Google's own write up notes that letting a flaky test count as failed only after three failures in a row reduces false alarms, but encourages developers to ignore flakiness in their own tests, and can delay the discovery of a real breakage10.

Example: one flake, start to finish

On a Tuesday the Fernhill Tea build turns red: the end to end test "a subscriber pauses for a month" failed. Priya reruns it on the same commit and it passes, and Playwright's report labels it flaky. She tags the test as quarantined, opens a ticket with her name and a deadline of Friday, and the release goes ahead on the remaining 17 browser tests plus all unit and integration tests. On Wednesday she runs the test 50 times in a row on her machine and sees 3 failures. Each failure happens when the test runs straight after "a customer cancels a subscription", which leaves a cancelled subscription in the shared test data. That is test order dependency. She makes each test create its own subscriber instead of reusing a shared one, runs it 200 times without a failure, removes the quarantine tag and closes the ticket.

Where does this test belong?

With all of that in place, choosing the level for a new test becomes a short series of questions. The first matters most: could a plain call to one function or object prove it? If so, write a unit test. If it needs the database, a mailer, a job or a fake of an outside service, write an integration test. Then ask separately whether a customer would lose an order or money if it broke in the browser. Only then does it earn a place in an end to end journey.

Tap or click to open full screen and move around

Example: three new requirements, routed

Fernhill Tea's owner asks for three changes in one week. Priya runs each through the questions.

  • "Gift subscriptions get a free tin with the first box." Deciding whether an order qualifies is a rule on plain values: is it a gift, is it the first box? Unit test. Would a customer lose an order if the tin were missing? No. No browser test.
  • "Send a reminder email three days before each subscription renews." This needs the database (which subscriptions renew when), a background job and the mailer. Integration test, with time moved forward in the test. Would a customer lose an order or money if it broke in the browser? There is no browser in it at all. No browser test.
  • "Customers in Northern Ireland can pick a new courier at checkout." The postage rule itself gets unit tests. Saving the choice on the order gets an integration test. And because a broken courier choice could stop a customer from paying, Priya extends the existing checkout journey with one extra step for a Belfast address, instead of writing a new end to end test.

Three requirements, five or six new unit and integration tests, one extended browser test, and the suite stays pyramid shaped.

What a business owner should ask

If you commission software rather than write it, you do not need to read the tests. You can still ask four questions that reveal a great deal:

  • Roughly how many tests are there at each level, and does the shape look like a pyramid or like an ice cream cone?
  • How long does the whole suite take, and does it run on every change?
  • How often does the build fail for no real reason, and what happens to a flaky test?
  • Which customer journeys are covered end to end, and which are the ones that must never break?

A good developer answers all four in a few minutes. The guide to TDD and BDD in Ruby on Rails goes deeper into the Rails side.

Conclusion: many fast tests, few slow ones, none you ignore

Test each behaviour at the lowest level that can prove it. Unit tests prove the rules, quickly and precisely, so you write many. Integration tests prove the parts work together, with fakes standing in for the services you do not own. End to end tests prove a person can complete the journeys that earn money, and because they are slow and fragile, you keep them few. Give each level a speed budget so the suite stays fast enough to run all the time. And treat a flaky test as a bug, because the arithmetic is unforgiving: a little randomness in each test makes a suite that nobody believes. The next chapter takes the most important of those journeys, the critical path, and shows how to find it, map it and guard it.

If you want to know whether your product's tests are a pyramid or an ice cream cone, and which journeys are not covered at all, book the free workshop: we look at your suite together, and the first two weeks of work are on me.

Sources

  1. 1.

    Adam Bender, Testing Overview, chapter 11 of Software Engineering at Google, edited by Titus Winters, Tom Manshreck and Hyrum Wright, O'Reilly, 2020, https://abseil.io/resources/swe-book/html/ch11.html

  2. 2.

    Rails Core Team, Testing Rails Applications, Ruby on Rails Guides, https://guides.rubyonrails.org/testing.html

  3. 3.

    Martin Fowler, TestPyramid, martinfowler.com, 1 May 2012, revised 15 November 2017, https://martinfowler.com/bliki/TestPyramid.html

  4. 4.

    Ham Vocke, The Practical Test Pyramid, martinfowler.com, 26 February 2018, https://martinfowler.com/articles/practical-test-pyramid.html

  5. 5.

    Kent C. Dodds, The Testing Trophy and Testing Classifications, kentcdodds.com, 3 June 2021, https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications

  6. 6.

    Martin Fowler, TestDouble, martinfowler.com, 17 January 2006, https://martinfowler.com/bliki/TestDouble.html

  7. 7.

    Martin Fowler, Mocks Aren't Stubs, martinfowler.com, 2 January 2007, https://martinfowler.com/articles/mocksArentStubs.html

  8. 8.

    Joseph Graves, Larger Testing, chapter 14 of Software Engineering at Google, edited by Titus Winters, Tom Manshreck and Hyrum Wright, O'Reilly, 2020, https://abseil.io/resources/swe-book/html/ch14.html

  9. 9.

    Stripe, Testing, Stripe documentation, https://docs.stripe.com/testing

  10. 10.

    John Micco, Flaky Tests at Google and How We Mitigate Them, Google Testing Blog, 27 May 2016, https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

  11. 11.

    Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, An Empirical Analysis of Flaky Tests, Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014, pages 643 to 653, https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf

  12. 12.

    Capybara contributors, Capybara README: Asynchronous JavaScript, GitHub, https://github.com/teamcapybara/capybara

  13. 13.

    Microsoft, Auto waiting (Actionability), Playwright documentation, https://playwright.dev/docs/actionability

  14. 14.

    Microsoft, Retries, Playwright documentation, https://playwright.dev/docs/test-retries