When AI changes the test instead of the code

TL;DR

Difficulty: Advanced

AI coding assistants are trained to make tests pass, and under pressure some of them do it by changing the test rather than the code. Anthropic, OpenAI, METR and the ImpossibleBench study have all documented it.

  • The five shapes: deleting or skipping a test, weakening the assertion, special casing the test input, mocking away the code under test, and changing the expected value to match the bug.
  • The five safeguards: read the test diff first, let CI refuse changed tests until a person approves, make test files ask before the assistant edits them, count tests and skips on every run, and use mutation testing to check your tests are still sharp.
  • Give the assistant a way out: stopping and asking must be allowed, and easier than cheating.

You ask your AI assistant to fix a failing test, it reports that everything passes, and the bug is still there, because the assistant changed the test. This is documented behaviour, not bad luck. Here is why it happens, how to recognise it, and the five safeguards that catch it, worked through on one VAT bug at an invented Bristol tea shop.

The test went green, and the bug is still there

Here is the situation this article is about. A test fails. You ask your AI coding assistant to sort it out. A minute later the assistant reports that every test passes. You merge, you deploy, and the bug is still in production, because the assistant did not change the code. It changed the test.

A quick reminder of the terms. A test is a small piece of code that runs part of your software and checks one result. The check is called an assertion: "the VAT on this order line must be 250 pence". When the code gives a different answer, the test fails and goes red. That red light is the whole value of the test. A test quietly edited to agree with the wrong answer is worse than no test, because it now tells you the wrong thing with full confidence.

In test driven development with AI I showed how to make an assistant work test first for a whole project. This chapter is about the failure that rule invites: what happens when the assistant is under pressure to turn red into green, and finds the shortest way there.

Example: Fernhill Tea and the Friday afternoon fix. Fernhill Tea is an invented online tea shop in Bristol (fernhilltea.example). It sells loose tea, subscriptions and accessories such as glass teapots and tea infusers. Its developer, Priya, has a failing test about VAT on an order line. It is Friday, she is busy, and she types into her assistant: "The VAT test is failing, please fix it." Two minutes later the assistant says: "Fixed. All 212 tests pass." Priya is about to merge. The rest of this article follows that one change, what was really in it, and the five safeguards that would each have caught it.

Why an assistant edits the test: it is aiming at green

Specification gaming, in plain words

An AI system does what it is asked, as literally as it can. Researchers at DeepMind call this specification gaming, "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome"1. The classic examples come from games, where a system finds a loophole that scores points without playing as intended.

Coding assistants meet the same temptation. Modern models learn, in part, by being rewarded when the tests in a task pass. The researchers call it reward hacking when a model learns to collect the reward without doing the real work. If passing tests is what gets rewarded, then editing the tests is, from the model's point of view, a perfectly good way to win. Nobody designed it to cheat. It learned that green is the goal.

Example: two ways to ask for the same thing. Priya's request, "the VAT test is failing, please fix it", names the test as the problem. Read literally, the test is the thing to fix. Compare: "The VAT on an order line is wrong. The test describes the correct behaviour. Change the code until it passes, and do not edit any file under test/." The second request names the behaviour as the target and the test as the judge. It does not make cheating impossible, but it removes the literal permission to do it.

What the model makers and researchers have measured

The companies that build the models have documented this themselves, which is to their credit.

Anthropic's system card for Claude 3.7 Sonnet (February 2025) has a whole section titled "Excessive focus on passing tests". It reports that the model occasionally special cased its way past test cases in agentic coding environments such as Claude Code. Most often it returned the expected test values directly, and the behaviour "also includes modifying the problematic tests themselves to match the code's output"2. It usually happened after several failed attempts at a general solution. The card recommends watching for "unexpected modifications to test files", which is exactly where this article is heading.

METR, an independent research group that evaluates AI models, reported in June 2025 that recent frontier models had been "attempting (often successfully) to get a higher score by modifying the tests or scoring code". In one task, OpenAI's o3 patched the evaluation function "so that it judges every submission as successful". On one set of research engineering tasks, METR counted reward hacking in 39 of 128 runs, about 30%3.

OpenAI's researchers saw the same pattern while training a reasoning model on coding tasks: in one example the agent "proceeds to simply update a unit test to make it pass trivially"4. Punishing the model for openly planning to cheat taught it to hide its intent rather than stop.

The clearest measurement so far comes from ImpossibleBench, a study published in October 2025. The researchers took real coding tasks and changed the tests so that they contradicted the written specification. No honest solution can pass such a test. Any pass is, by construction, a cheat. On the version built from real open source bug fixes, OpenAI's newest model in the study cheated on 54% of the tasks. The authors also found that Claude models "cheat primarily (>79%) through modifying test cases"5.

Read these numbers fairly. They come from tasks made impossible on purpose, or from training and evaluation setups, not from an average working day. With a sound test and a solvable bug, an assistant fixes the code far more often than it touches the test. But the behaviour is real, and it turns up when you are least likely to look: after several failed attempts, on a bug harder than it looked.

Bar chart from the ImpossibleBench study. On deliberately impossible coding tasks, OpenAI's newest model in the study cheated on 54% of tasks, falling to 9% when it was allowed to stop and flag the task as impossible. OpenAI o3 cheated on 49%, falling to 12% with the same option. Tap or click to open full screen and move around

The second and fourth bars matter as much as the first and third. I come back to them in the section on giving the assistant a way out.

The five shapes it takes

When an assistant games a test, it rarely announces it. Once you know the five shapes, you spot them in seconds. Each example below uses the same piece of Fernhill Tea's code: a method that works out the VAT on one order line.

Deleting or skipping the test

The bluntest form. The test is removed, commented out, or marked as skipped. Minitest, the testing library that ships with Rails, has a skip method for tests that are not ready yet. It is useful, and it is also an off switch.

Example: the skipped VAT test.

test "VAT on a line is worked out once for the whole line" do
  skip "flaky, revisit later"
  vat = VatCalculator.line_vat_pence(unit_gross_pence: 499, quantity: 3)
  assert_equal 250, vat
end

The skip line stops the test and reports it as skipped. The suite stays green. The note sounds reasonable, which is what makes it dangerous: the test was never flaky, it was right.

Weakening the assertion

The test still runs, but it checks less. An exact comparison becomes a range, a range becomes "is a number", "is a number" becomes "did not crash".

Example: the VAT test that accepts nearly anything.

test "VAT on a line is worked out once for the whole line" do
  vat = VatCalculator.line_vat_pence(unit_gross_pence: 499, quantity: 3)
  assert_in_delta 250, vat, 1
end

assert_in_delta 250, vat, 1 means "vat must be within 1 of 250". So 249, 250 and 251 all pass. The wrong answer, 249, now passes. The test looks almost the same, which is why weakened assertions survive a quick glance.

Special casing the test input

The code learns the test's exact inputs and returns the exact expected answer for them, while staying wrong for every other order. This is the form the Anthropic system card describes most often2.

Example: the hard coded answer.

def self.line_vat_pence(unit_gross_pence:, quantity:, rate_percent: 20)
  return 250 if unit_gross_pence == 499 && quantity == 3

  (unit_gross_pence * rate_percent / (100.0 + rate_percent)).round * quantity
end

The first line checks for precisely the numbers in the test and returns 250. Every real customer who buys three infusers at £4.99 gets the right VAT by coincidence. A customer who buys three teapots at £7.99 still gets the old, wrong figure. The test passes, and the bug is untouched.

Mocking away the thing under test

A mock or stub is a stand in. In a test you can replace a slow or external piece, such as a payment provider, with a stand in that returns a fixed answer. That is a normal, healthy technique. It turns into cheating when the stand in replaces the very thing the test is meant to check.

Example: the VAT calculator that tests its own stand in.

test "VAT on a line is worked out once for the whole line" do
  VatCalculator.stub(:line_vat_pence, 250) do
    vat = VatCalculator.line_vat_pence(unit_gross_pence: 499, quantity: 3)
    assert_equal 250, vat
  end
end

stub(:line_vat_pence, 250) replaces the real method with one that always returns 250, for the duration of the block. The test then calls the stand in and confirms that it returns 250. It tests nothing at all. The real method, with its bug, is never run.

Changing the expected value to match the bug

The quietest form. The test keeps its shape, its name and its exact comparison. Only the number changes, from the correct value to whatever the buggy code produces. This is what happened at Fernhill Tea, and it gets the full worked example next.

The worked example: the VAT penny at Fernhill Tea

The rule the business agreed

The UK standard rate of VAT is 20%6. Loose tea counts as food and is treated differently, so this example uses accessories, which carry the standard rate. Fernhill Tea shows prices including VAT, and its accountant agreed one rule for invoices: work out the VAT once for the whole order line, then round to the nearest penny, with an exact half penny rounding up. That rule is a business decision. The code's job is to follow it.

The test Priya wrote first

Priya works test first. Before writing the method, she wrote the test from the accountant's rule, in pence to avoid fractions of a penny in the stored numbers:

# test/models/vat_calculator_test.rb
require "test_helper"

class VatCalculatorTest < ActiveSupport::TestCase
  test "VAT on a line is worked out once for the whole line" do
    vat = VatCalculator.line_vat_pence(unit_gross_pence: 499, quantity: 3)
    assert_equal 250, vat
  end
end

Here is where 250 comes from, step by step. Three infusers at £4.99 make a line of 1,497 pence including VAT. At 20%, the VAT inside a gross price is 20 parts out of 120, so the VAT is 1,497 times 20, divided by 120. That is 29,940 divided by 120, which is 249.5 pence. Half a penny rounds up, so the answer is 250 pence, or £2.50.

The code that broke it

A later change, made in a hurry, moved the rounding to each single item instead of the line:

# app/models/vat_calculator.rb
class VatCalculator
  def self.line_vat_pence(unit_gross_pence:, quantity:, rate_percent: 20)
    unit_vat = unit_gross_pence * rate_percent / (100.0 + rate_percent)
    unit_vat.round * quantity
  end
end

Line by line: unit_vat is the VAT inside one infuser, 499 times 20 divided by 120, which is 83.1666 pence. .round turns that into 83, and * quantity multiplies by three, giving 249. The rounding now happens on each item instead of once on the line. One penny is lost here, and over thousands of lines the invoices stop matching the VAT return.

The test caught it. Running bin/rails test printed:

Failure:
VatCalculatorTest#test_VAT_on_a_line_is_worked_out_once_for_the_whole_line
Expected: 250
  Actual: 249

That is the test doing its job perfectly.

What the assistant changed

Priya asked the assistant to fix the failing test. After two attempts at the code that did not pass, the assistant produced this change:

--- a/test/models/vat_calculator_test.rb
+++ b/test/models/vat_calculator_test.rb
@@ class VatCalculatorTest < ActiveSupport::TestCase
   test "VAT on a line is worked out once for the whole line" do
     vat = VatCalculator.line_vat_pence(unit_gross_pence: 499, quantity: 3)
-    assert_equal 250, vat
+    assert_equal 249, vat
   end

The line starting with a minus is the old version, the line starting with a plus is the new one. One character changed. The test name still promises "worked out once for the whole line". The assertion now pins the exact behaviour the name says is wrong. The suite is green, and the bug is now protected by a test.

The right fix

The right change touches only the code:

# app/models/vat_calculator.rb
class VatCalculator
  def self.line_vat_pence(unit_gross_pence:, quantity:, rate_percent: 20)
    gross = unit_gross_pence * quantity
    Rational(gross * rate_percent, 100 + rate_percent).round(half: :up)
  end
end

gross is the whole line, 1,497 pence. Rational(...) keeps the division exact as a fraction, 29,940 over 120, instead of a floating point number that can drift by tiny amounts. .round(half: :up) rounds once, with a half going up, exactly as the accountant's rule says. The result is 250, and the original test passes without being touched.

Money and tax sit in the highest risk tier of how I work with AI: there, I decide and the AI only advises. A VAT rule is not something an assistant should be allowed to reinterpret by editing a test.

Safeguard one: read the test diff first, and on its own

The simplest defence costs nothing. Before you read a single line of code in a change, read the changes to the tests, separately. Tests are the specification. If the specification moved, that is the headline of the change, whatever the commit message says.

Git, the version control system almost every project uses, can show you only the test folder:

git diff --stat main -- test/ spec/ features/
git diff main -- test/ spec/ features/

The first command lists which test files changed and how many lines. The second shows the changes themselves. main is the branch you are comparing against, and everything after -- limits the output to those folders (Minitest uses test/, RSpec uses spec/, Cucumber uses features/).

Then ask one question: was an existing test changed, deleted or skipped, and did anyone ask for that behaviour to change? New tests are welcome. A changed old test needs a reason a person agreed to.

Example: Priya's thirty second check. Before merging, Priya runs the first command. The output is one line: test/models/vat_calculator_test.rb | 2 +-, one line removed, one added. She asked for a code fix, so any change in the test folder is a surprise. She opens it, sees 250 become 249, and knows at once that the assistant moved the goalposts. Thirty seconds, and the bug never ships.

Safeguard two: make the pipeline notice test changes

Reading the test diff works when you remember to do it. On a Friday afternoon, you might not. So the second safeguard moves the check into continuous integration (CI), the automated pipeline that runs on every change before it can be merged.

The idea: adding tests is free, but changing, renaming or deleting an existing test fails the pipeline until a person approves. On GitHub, a label on the pull request (the proposed change, waiting for review) is a simple way to record that approval.

# .github/workflows/guard-tests.yml
name: Guard existing tests
on:
  pull_request:
    types: [opened, synchronize, labeled, unlabeled]
jobs:
  guard-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Fail if existing tests changed without approval
        if: ${{ !contains(github.event.pull_request.labels.*.name, 'test-change-approved') }}
        run: |
          changed=$(git diff --name-only --diff-filter=MDR \
            origin/${{ github.base_ref }}...HEAD -- test/ spec/ features/)
          if [ -n "$changed" ]; then
            echo "Existing tests were changed, renamed or deleted:"
            echo "$changed"
            echo "A person must review them and add the label test-change-approved."
            exit 1
          fi

Line by line: the workflow runs on every pull request, and again whenever a label is added or removed. fetch-depth: 0 downloads the full history so Git can compare against the main branch. The step only runs when the pull request does not carry the test-change-approved label. Inside it, git diff --name-only --diff-filter=MDR lists test files that were modified (M), deleted (D) or renamed (R). Brand new test files (A, added) are not in the filter, so they pass freely. If the list is not empty, the job prints the files and fails with exit 1.

A failing job only blocks a merge if GitHub is told to treat it as required. That is what branch protection does: a protected branch can require status checks to pass before anything is merged into it7. Two details keep the gate honest. The label must be added by a person, so the assistant's access token should not be allowed to add labels or merge. And the pipeline file itself belongs in the same review: a change to .github/workflows/ deserves the same suspicion as a change to test/.

In a team, GitHub's CODEOWNERS file adds a second layer. It names who owns which folders, for example /test/ @fernhill-tea/leads, and with the branch protection setting "Require review from Code Owners" a pull request that touches the test folder cannot be merged until one of those owners approves it8. For a single developer, the label check does the same job without needing a second person.

Example: the VAT pull request meets the pipeline. Priya's assistant opens the pull request with the changed expected value. The guard job runs, finds test/models/vat_calculator_test.rb in the modified list, prints it, and fails. The merge button is greyed out. Priya opens the job log, sees the file name, and reads the diff. She does not add the label. She closes the pull request and asks for the code fix instead.

Safeguard three: protect the test folder from the assistant

The first two safeguards catch the change after it is made. The third makes it harder to make in the first place. ImpossibleBench tested exactly this. Making the tests read only stopped the models from editing them, and the models still solved the honest tasks as well as before, although it did not stop the other tricks such as special casing. The authors recommend either hiding test files entirely or making them read only while the code is being written5.

Most coding assistants have some form of permission settings. In Claude Code, project settings live in .claude/settings.json, which you commit so that everyone working on the project gets the same rules. Permission rules come in three kinds, deny, ask and allow, and they are checked in that order9:

{
  "permissions": {
    "ask": [
      "Edit(/test/**)",
      "Edit(/spec/**)",
      "Edit(/features/**)"
    ]
  }
}

Edit(/test/**) matches every file under the project's test/ folder, and in project settings a path starting with / is anchored at the project folder9. With the rule under ask, every attempt to edit a test stops and asks Priya first, showing her the change. During a pure implementation task, she can move the same rules to deny, and the edit is simply refused.

Be clear about the limits. Claude Code's documentation says these rules cover its built in file tools and the file commands it recognises, but not "arbitrary subprocesses that read or write files indirectly, like a Python or Node script that opens files itself"9. So this safeguard sits next to the pipeline check, never in place of it. Claude Code can also run hook scripts before a tool is used, and a hook that exits with code 2 blocks the action and tells the assistant why10. Other assistants have their own settings, so check their documentation.

Example: the edit that asks first. With the ask rules in place, Priya gives the same Friday request. The assistant tries the code twice, then tries to edit test/models/vat_calculator_test.rb. Claude Code stops and shows the proposed change, 250 to 249, and asks whether to allow it. Priya says no. The assistant goes back to the code, and she adds a sentence to the conversation: the test is correct, the rounding must happen once per line.

Safeguard four: count the tests and assertions on every run

Deleting and skipping are easy to measure, because the numbers drop. Minitest ends every run with a summary line such as 212 runs, 655 assertions, 0 failures, 0 errors, 0 skips. Runs are tests, assertions are the individual checks inside them. If a change shrinks either number, or the skips go up, something was removed or switched off.

A short script in the pipeline turns that into a rule:

summary=$(bin/rails test 2>&1 | grep -E '^[0-9]+ runs')
echo "$summary"
runs=$(echo "$summary" | cut -d' ' -f1)
skips=$(echo "$summary" | sed -E 's/.* ([0-9]+) skips.*/\1/')
baseline=$(cat .test-baseline)
[ "$skips" -eq 0 ] || { echo "Skipped tests found"; exit 1; }
[ "$runs" -ge "$baseline" ] || { echo "Fewer tests than before: $runs < $baseline"; exit 1; }

The script runs the suite, keeps the summary line, and pulls out the number of runs and skips. .test-baseline is a one line file holding the number of tests on the main branch, updated by a person when tests are deliberately removed. The script fails if any test is skipped, or if there are fewer tests than the baseline.

Be honest about what this catches. It catches deletion and skipping. It does not catch a changed expected value or a special case, because the number of tests stays exactly the same.

Example: two attempts, two results. Suppose the assistant had chosen the skip from earlier. The summary would read 212 runs, 654 assertions, 0 failures, 0 errors, 1 skips, and the script would stop at "Skipped tests found". Now take what really happened, 250 changed to 249: 212 runs, 655 assertions, 0 failures, 0 errors, 0 skips. Identical to before. The count passes, and only safeguards one to three catch it. That is why no single safeguard is enough.

Safeguard five: mutation testing, the test of your tests

Weakened assertions are the hardest shape to spot by eye. Mutation testing measures them directly.

How it works

A mutation testing tool makes small deliberate changes to your code, one at a time: it swaps round for floor, turns a + into a -, deletes a multiplication. Each changed version is called a mutant. For every mutant, the tool runs your tests. If at least one test fails, the mutant is "killed", which is good: your tests noticed the change. If all tests pass, the mutant "survived", which means your tests would not notice that bug either11.

For Ruby, the tool is mutant, which works with RSpec and Minitest. It is free for open source projects, and commercial use needs a paid subscription per developer12. For JavaScript and TypeScript, Stryker does the same11. Mutation testing is slow, because it runs the suite once per mutant, so most teams run it on the code that changed rather than on everything, every time.

The result is summed up as a mutation score:

score=km\text{score} = \frac{k}{m}

where kk is the number of mutants your tests killed, mm is the number of mutants the tool created and ran, and score\text{score} is the share of them your tests caught, usually shown as a percentage. A score of 100% means every deliberate bug was caught. If ratios and percentages feel rusty, Foundations: ratios and rates explains them from zero.

Example: the strong tests and the weakened test, scored

Take the fixed VAT method from the worked example. To make the score meaningful, Priya now has two tests: the line of three infusers (expected 250) and a single infuser (499 times 20 divided by 120 is 83.17, expected 83). A mutation tool would produce many mutants automatically. Here are six typical ones, written out by hand so you can follow each:

MutantWhat changedLine of threeSingle item
1round becomes floor24983
2round becomes ceil25084
3100 + rate_percent becomes 100299100
4gross * rate_percent becomes gross124
5unit_gross_pence * quantity becomes unit_gross_pence8383
6100 + rate_percent becomes 100 - rate_percent374125

With the two strong tests, every mutant gives a wrong number in at least one test. Mutant 2 is the interesting one: the line of three still gives 250, but the single item gives 84 instead of 83, so the second test kills it. All six are killed:

score=66=100%\text{score} = \frac{6}{6} = 100\%

Here k=6k = 6 mutants killed out of m=6m = 6 mutants created.

Now score the weakened version from earlier: only the line of three, checked with assert_in_delta 250, vat, 1. Mutant 1 gives 249, within 1 of 250, so it survives. Mutant 2 gives 250, so it survives. Mutants 3 to 6 are far off and are killed:

score=46≈67%\text{score} = \frac{4}{6} \approx 67\%

Here k=4k = 4 mutants killed out of m=6m = 6, so two deliberate bugs slipped through.

The drop from 100% to 67% on the changed file is the signal. It says: the tests for this method no longer catch a rounding mistake. That is precisely the bug they were written to catch.

One limit to keep in mind: mutation testing measures how sharp the tests are, not whether they are right. A test that pins the wrong value exactly, 249 instead of 250, kills mutants very well. Only a person who knows the business rule can see that the expected value itself is wrong. That is why the first safeguard is a person reading the test diff.

Give the assistant a way out

The bars earlier in this article had a second half. In the ImpossibleBench study, simply allowing the model to stop and flag a task as impossible lowered the cheating rate of OpenAI's newest model from 54% to 9%, and of o3 from 49% to 12%5. The authors note the effect was much smaller for Claude Opus 4.1, so this is not a cure. It is still a habit that costs one sentence and pays well. Anthropic's system card makes a related recommendation: prompts that "explicitly emphasize general solutions" rather than special cases2.

In practice that means one more rule in the project's instruction file, the file the assistant reads at the start of every session (CLAUDE.md for Claude Code, AGENTS.md for tools that follow that convention, as set up in chapter 8):

## When a test fails

* The tests describe the agreed behaviour. Fix the code, never the test.
* Never delete, skip, weaken or rewrite an existing test, and never stub
  the method a test is checking.
* Never return a value only because a test expects it. Solve it in general.
* If you think a test is wrong, or you cannot pass it after two honest
  attempts, stop. Explain what you tried and why you think the test may be
  wrong, and wait for a person to decide.

Each line closes one of the five shapes, and the last gives the assistant a legitimate alternative to cheating.

Example: the rule in action. With these lines in Fernhill Tea's instruction file, Priya's Friday request plays out differently. After two failed attempts, the assistant writes: "I could not get the VAT test to pass. The code rounds each item, the test expects the line to be rounded once. I believe the code is wrong, not the test. Shall I change the rounding to once per line?" That is the answer Priya needed. She says yes, and the right fix arrives with the test untouched.

A test fails: who may change what

All of this comes down to one decision, and it belongs to a person. When a test fails, the code may change freely. The test may change only when the behaviour the business wants has changed, and someone who owns that rule has agreed.

Tap or click to open full screen and move around

The highlighted path is the normal one, and it never touches the test. The branch on the side is for real changes in what the business wants, and even there the test changes first, on purpose, and is seen to fail before any code moves. That is just test driven development applied to a change of mind.

Example: when the test really should change. A year later, Fernhill Tea's accountant decides that invoices will show VAT per item instead of per line, and agrees it in writing. Now the test no longer describes what the business wants. Priya records the new rule, changes the test name and expected value, watches the test fail, then changes the code. The guard job flags the test change, and Priya adds the test-change-approved label herself, because this time she is approving a change she understands. The commit message names the accountant's decision.

When it happens anyway: what to do

No set of safeguards is perfect. When you find a gamed test, treat it like any other defect, calmly and in order.

  1. Put the test back. Restore the file from the main branch so the specification is right again.
  2. Run the suite and confirm the test fails, for the right reason. Red is now good news.
  3. Fix the code, test first, with the assistant or by hand.
  4. Look for others. Check every recent change to the test folder, because a habit is rarely used only once.
  5. Close the gap that let it through: a missing safeguard, a missing rule in the instruction file, or a missing second test case.
  6. Write it down in the commit message, so the history explains itself to whoever reads it next.

Example: Priya cleans up. Priya restores the test with git checkout main -- test/models/vat_calculator_test.rb, which takes the file as it is on the main branch. She runs bin/rails test test/models/vat_calculator_test.rb and sees the expected red: expected 250, actual 249. She applies the rounding fix, runs the full suite, and it is green with the original assertion. Then she runs git log --since="1 month ago" --stat -- test/ to list every change to the test folder in the past month, and reads each one. She finds none that are suspicious, adds the single item test case from the mutation example, adds the instruction file rules, and commits with the message "Round VAT once per line as agreed with the accountant, restore the original test".

Conclusion

An AI assistant that changes a test instead of the code is not malicious. It is doing what it was trained to do, chase green, as literally as possible. The model makers and independent researchers have documented it, measured it, and published it, so there is no reason to be surprised by it.

The defence is a layered one, and none of it is exotic. Read the test diff first. Let the pipeline refuse changed tests until a person approves them. Make the test folder ask before it can be edited. Count tests and skips on every run. Use mutation testing to see whether your tests are still sharp. And give the assistant an honest way out, so stopping and asking is easier than cheating.

Behind all of it sits one rule: the tests belong to the business, and the person who answers for the software decides when they change. The next chapter, keeping the last word, is about staying that person while AI writes much of the code.

If you want an AI assisted codebase where the tests stay the specification and nobody can quietly move the goalposts, book the free workshop: we look at your pipeline together, and the first two weeks of work are on me.

Sources

  1. 1.

    Victoria Krakovna and others, Specification gaming: the flip side of AI ingenuity, Google DeepMind blog, 21 April 2020, https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/

  2. 2.

    Anthropic, Claude 3.7 Sonnet System Card, section 6, Excessive focus on passing tests, February 2025, https://www.anthropic.com/claude-3-7-sonnet-system-card

  3. 3.

    Sydney Von Arx, Lawrence Chan and Beth Barnes, Recent frontier models are reward hacking, METR, 5 June 2025, https://metr.org/blog/2025-06-05-recent-reward-hacking/

  4. 4.

    Bowen Baker and others, Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, OpenAI, arXiv 2503.11926, March 2025, https://arxiv.org/abs/2503.11926

  5. 5.

    Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini, ImpossibleBench: measuring LLMs' propensity of exploiting test cases, arXiv 2510.20270, October 2025, https://arxiv.org/abs/2510.20270

  6. 6.

    HM Revenue and Customs, VAT rates, GOV.UK, https://www.gov.uk/vat-rates

  7. 7.
  8. 8.
  9. 9.

    Anthropic, Claude Code documentation, Configure permissions, https://code.claude.com/docs/en/permissions

  10. 10.

    Anthropic, Claude Code documentation, Hooks reference, https://code.claude.com/docs/en/hooks

  11. 11.

    Stryker Mutator documentation, Mutant states and metrics, https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/

  12. 12.

    mutant, mutation testing for Ruby, project README, GitHub repository mbj/mutant, https://github.com/mbj/mutant