Will AI like ChatGPT ever replace developers? A developer's view, with the evidence

The short answer: an assistant, not a replacement

People ask me this more than almost anything else. Will tools like ChatGPT replace developers? My answer, as someone who writes code for a living, has not changed since I first wrote this piece: AI is an invaluable assistant for developers, but it cannot replace the need for human expertise in coding. It does not truly grasp the nuances of a problem the way a developer does.

That is exactly the opinion you would expect a developer to hold. So I put it next to the best evidence there is: not vendor marketing, but controlled studies where one group of developers gets the AI tool, a comparable group does not, and someone measures the difference.

Some of those studies show developers finishing much faster with AI. One careful study showed experienced developers getting slower while feeling faster. Both can be true, and the reason why is the most useful thing a business owner can learn about handing work to these tools.

One sibling question I do not repeat here: if AI writes the code, who answers for it when it breaks? I cover that, with the security research and the dangers in the order they get expensive, in AI writes the code. Who is responsible for it?

Throughout, I use one invented business. Butterslip Larder is a small online deli in Kilkenny, run by Amira, selling Irish farmhouse cheese, charcuterie and gift boxes from butterslip.example. Nobody real, every number made up so that I can show every step.

How a controlled study measures productivity

Before the numbers, it helps to know how they are produced, because the method decides what a number means.

In a randomised controlled trial, developers (or tasks) are split by chance into two groups. One group may use the AI tool, the other may not. Because the split is random, the two groups are alike on average in skill, experience and luck, so a difference in the outcome can be put down to the tool. The outcome is usually time to finish a task, or the number of tasks finished in a week.

Two things are worth checking every time you read such a study. What was the task? A small, well specified exercise is very different from a change to a large codebase someone has maintained for years. And what was measured? Speed is not the same as correctness, and finishing more tasks is not the same as finishing them well.

Worked example: reading the GitHub Copilot experiment

The best known early study came from researchers at Microsoft Research, GitHub and MIT in 2023. They recruited 95 professional developers through an online jobs platform and split them at random: 45 could use GitHub Copilot, an AI tool that suggests code as you type, and 50 could not, though the control group was free to search the web and Stack Overflow. Everyone had to build the same thing: a small web server in JavaScript, as fast as possible, checked by a test suite of 12 checks.¹

Among those who finished, the Copilot group took 71.17 minutes on average and the control group 160.89 minutes. The reduction in time is one minus the ratio of the two:

$$1 - \frac{71.17}{160.89} = 1 - 0.442 = 0.558$$

Here $71.17$ is the average time with the tool in minutes, $160.89$ the average without, and $0.558$ the share of time saved, 55.8%. The paper gives a 95% confidence interval of 21% to 89%, which means the true saving in that setting could plausibly be anywhere in that wide range. The same paper found that less experienced developers benefited most.¹ How to read an interval like that is set out in Foundations: confidence intervals.

For Amira, it says that on one small, clearly specified task written from nothing, the tool roughly halved the time. It says nothing about changing the checkout of a deli that has traded for six years. Several authors worked for the company behind the tool, which is worth knowing.

The field experiments: more tasks finished, measured inside real companies

A second, much larger study moved out of the lab. Researchers from Princeton, MIT, Wharton and Microsoft analysed three randomised trials that Microsoft, Accenture and an anonymous Fortune 100 electronics manufacturer had run during their ordinary course of business. A random subset of developers got access to an AI coding assistant, and the trials lasted between two and eight months before everyone got it.²

Pooled across 4,867 developers, the authors estimate a 26.08% increase in completed tasks per week among developers who used the tool, with a standard error of 10.3%. Each experiment on its own was noisy. Less experienced developers adopted the tool more and gained more.²

The unit of "completed task" was the pull request, the bundle of code a developer submits for review by a colleague. That is a measure of output. It is not a measure of whether the code was right, which the authors acknowledge: if developers accept suggestions without reviewing them, quality could fall.²

Worked example: what 26% more pull requests would mean for Amira

Suppose Amira's developer finishes 10 pull requests in a typical month. A 26% increase would take that to

$$10 \times 1.26 = 12.6$$

Here $10$ is the monthly pull requests without the tool, $1.26$ is one plus the estimated increase of 26%, and $12.6$ is the expected monthly pull requests with it. The standard error of about 10 points means the true effect could easily be half that or noticeably more, a point explained in Foundations: sampling.

Two or three extra pieces of work a month is a fair gain, and a long way from "you no longer need the developer". Every one of them still goes through review by a person.

The study that found experienced developers slower

In July 2025 METR, a research organisation that studies AI capabilities, published the result that surprised almost everyone. It ran a randomised trial with 16 experienced open source developers working on 246 real tasks in mature projects they knew well. Each task was randomly assigned to "AI allowed" or "AI not allowed". The tools were the ones available between February and June 2025.³

With AI allowed, tasks took 19% longer.³ METR's later post gives the confidence interval as 2% to 39% longer.⁴ These were people who knew their own codebases deeply, where a general tool has the least to add and the most to get subtly wrong.

Worked example: a ten hour task, three ways

The striking part is not the slowdown alone. Before the study, the developers forecast that AI would cut their completion time by 24%. After it, they estimated it had cut their time by 20%.³ Take a task that would take 10 hours without AI:

$$\begin{aligned} \text{forecast} &= 10 \times (1 - 0.24) = 7.6 \text{ hours} \\ \text{felt} &= 10 \times (1 - 0.20) = 8.0 \text{ hours} \\ \text{measured} &= 10 \times (1 + 0.19) = 11.9 \text{ hours} \end{aligned}$$

In each line $10$ is the hours the task takes without AI, $0.24$ and $0.20$ are the forecast and felt reductions, and $0.19$ is the measured increase. The gap between how it felt and what happened is $11.9 - 8.0 = 3.9$ hours on a single task. Percentages like these are explained in Foundations: percentages.

The horizontal axis shows four views of the same work: the time without AI, what the developers expected, what they believed afterwards, and what was measured. The vertical axis is task time, with the time without AI set to 100, so lower is faster. The takeaway is the distance between the third and fourth bars. Your own sense of being sped up is not evidence, and neither is your developer's, mine included.

The evidence moves fast, and the newest is honest about its limits

A study from early 2025 is not the last word, and METR says so itself. On 24 February 2026 it published an update on a larger follow up that began in August 2025: 57 developers (10 returning, 47 new), 143 repositories and over 800 tasks.⁴

The raw estimates leaned the other way. Returning developers' tasks took an estimated 18% less time with AI, with an interval from 38% less to 9% more. New developers' tasks took 4% less, from 15% less to 9% more. Both intervals include zero.⁴

METR then explained why it does not trust these numbers as a clean estimate. Many developers now declined to take part, or declined to submit tasks they did not want to do without AI, so the work most suited to AI was underrepresented. Time tracking became unreliable when people ran several AI agents at once. METR concluded that the central estimate is "likely a bad proxy" for the real effect, while saying it believes developers were likely more sped up in early 2026 than in early 2025.⁴

Worked example: what Amira should take from a moving target

Amira has 56% faster on a small fresh task, 26% more pull requests inside large companies, 19% slower for experts in mature code, and a 2026 update leaning towards faster whose authors will not stand behind the size. She does not need a winner. She needs one rule: the answer depends on the task. Fresh, well specified, low stakes work is where every study finds the tool helps. Changes deep inside code that already works, where every line has a reason, is where the gains are smallest and least certain. That is the split she will use later to decide what to hand over.

What developers themselves report

Controlled studies measure time. Surveys show where developers feel the time goes.

In the Stack Overflow Developer Survey 2025, 84% of respondents said they used or planned to use AI tools in their development process, and 50.6% of professional developers used them daily. Yet more distrusted the accuracy of AI output than trusted it: 45.7% against 32.7%. The most common frustration, named by 66%, was "AI solutions that are almost right, but not quite", and 45.2% said debugging AI generated code is more time consuming.⁵

The 2026 survey, with 30,903 responses, asked the questions differently, so the figures are not directly comparable. Over 70% of respondents now use AI daily. Trust has become conditional: 48% say they trust AI when they can easily validate its answers.⁶

That matches my own experience closely. I have tested this myself, and at times I have spent more time working with AI to debug and refactor code than if I had just written it myself. You end up running in circles. AI suggests changes, but sometimes those changes create new issues or skip over things that were working fine. "Almost right" is the most expensive kind of wrong, because it passes a glance. I wrote about what that costs when it reaches production in the €400+ an hour AI disaster cleanup business.

Worked example: when easy to validate is the whole question

The 2026 trust figure hides a practical test. Amira asks for two things. First, a list of 40 sample products to fill her test shop. She can check that at a glance: is it 40 rows, do they look like deli products. Second, a change to how gift box prices are calculated when two discounts apply. She cannot check that at a glance, because the numbers look plausible either way. The first is easy to validate, so AI is a fine author. The second is not, so whoever accepts it needs to understand it line by line.

Where AI earns its keep

I am not a sceptic who refuses the tools. ChatGPT and its peers are a sound help, and in my experience they are good at:

• generating repetitive code, such as seed files for a database or boilerplate
• suggesting syntax or library alternatives
• offering ideas for solving common coding problems
• helping with documentation and comments

They speed things up most where you are less familiar with a syntax or a library, or when you want an approach for a feature in a framework you do not use every day. In my experience it can get you around 80% of the way there. It is like having a junior developer or a mentor beside you with ideas and approaches. This lines up with both the Copilot experiment and the field experiments, where less experienced developers gained the most.¹²

Worked example: seed data for the Butterslip Larder test shop

A seed file fills a development database with made up records so the shop can be tested without real customers. Repetitive, low risk and easy to check: the textbook AI task. Amira's developer asks for 40 products and a few test customers and gets something like this:

# db/seeds.rb, test data only, never run against production
products = [
  { name: "Slieveardagh soft cheese, 200g", price: 6.50, category: "Cheese" },
  { name: "Smoked Coppa, 100g", price: 5.25, category: "Charcuterie" },
  { name: "Kilkenny Gift Box", price: 45.00, category: "Gifts" }
  # ... 37 more, generated
]
products.each { |attrs| Product.find_or_create_by!(attrs) }

5.times do |i|
  Customer.find_or_create_by!(email: "test-customer-#{i}@butterslip.example")
end

The developer still reads it: test data only, a reserved example domain so no real person is ever emailed, records created only if missing. That check takes a minute, no bother, because the code is short and the stakes are low.

Where the developer stays essential: edge cases

AI does not truly grasp the nuances of a problem. If I am building a complex feature for a shop, say a custom checkout with various conditional discounts, AI might generate the initial function well. But it will often miss edge cases, such as handling failed payments or calculating discounts correctly across several products. A developer considers the full picture, the edge cases and the potential for bugs down the line.

Worked example: the discount that looks right

Amira runs two offers at once. A code, KILKENNY5, takes €5 off the order. And every order over €50 gets 10% off. Her rule, which is a business decision and could reasonably have gone the other way, is that the 10% is worked out on the goods before the code, and the code comes off at the end.

A customer buys a cheese box at €30 and a charcuterie box at €25, so goods of €55. The correct total under Amira's rule:

$$55 - (0.10 \times 55) - 5 = 55 - 5.50 - 5 = 44.50$$

Here $55$ is the goods value in euro, $0.10$ is the 10% discount, $5.50$ is that discount in euro, and $5$ is the code. The customer pays €44.50.

A plausible AI draft applies the €5 code to each line item instead of the order, then takes 10% off what is left:

$$55 - (2 \times 5) - 0.10 \times (55 - 10) = 55 - 10 - 4.50 = 40.50$$

Here $2 \times 5$ is the code wrongly applied to both lines, and $0.10 \times (55 - 10)$ is the percentage worked out on the reduced amount. The customer pays €40.50, and the shop loses €4 on the order. Nothing crashes. No error appears. On 300 such orders a month that is $300 \times 4 = 1{,}200$ euro a month, found only when someone reconciles the takings.

The code looked right and would have survived a quick demo. What catches it is a developer who knows the rule and writes it down as a test first.

Tests first, because they force the thinking

What many people do not realise is that practices like Test Driven Development (TDD) and Behaviour Driven Development (BDD) are essential for writing quality, maintainable code. TDD means writing the test before the code, which pushes you to think about what the feature must do and what could go wrong. BDD describes behaviour in plain language, so the code is understandable by people who are not developers and lines up with the business goals.

Can ChatGPT help with this? Absolutely. I often use it for inspiration when I am stuck on a test or looking for coverage ideas. But at the end of the day it is still me who must write the test and consider the full scope of the feature. Writing the test forces me to think about error handling, edge cases and performance, and that thinking is the part I am responsible for.

Worked example: pinning Amira's discount rule

The developer writes this before the discount code exists:

RSpec.describe Checkout do
  it "takes 10% off goods over €50, then the KILKENNY5 code once per order" do
    order = build_order(lines: [30.00, 25.00], code: "KILKENNY5")
    expect(Checkout.new(order).total).to eq(44.50)
  end

  it "keeps the basket and shows a message when the payment fails" do
    order = build_order(lines: [30.00], payment: :declined)
    result = Checkout.new(order).complete
    expect(result.status).to eq(:payment_failed)
    expect(order.reload.lines.count).to eq(1)
  end
end

The first test encodes Amira's rule with the exact figure, €44.50. The AI draft that charged €40.50 fails it at once, before any customer sees it. The second covers the edge case that a happy path demo never shows: a declined card must not empty the basket. Now AI can draft the implementation as often as it likes. The tests, written by a person who understood the business, decide whether it is right.

The large codebase problem

AI tools can certainly help with snippets, or with debugging small issues. But to understand a large codebase and integrate a new feature, you need deep familiarity with its syntax, architecture and design patterns. Nothing replaces a developer critically engaging with the code and understanding what every line is doing.

Without that understanding, AI help leads to incorrect assumptions or unfinished solutions: output that looks right on the surface, and no way to pinpoint why it fails in the context of the rest of the project. Adding a payment gateway, for example, means understanding how the application talks to the gateway, the data flow and the security implications. AI can generate the code. It does not bring the contextual awareness or the security foresight.

When I first wrote this article, one of the biggest limitations I found with ChatGPT was that it did not handle large blocks of code well: long input got cut off or was not processed properly. The tools change fast, so I would not hold that specific limit against them today. The lesson outlasts it. This is also the setting of the METR result, experienced developers in mature code, where the measured gain was smallest.³ How fast code built without that understanding piles up into an architecture nobody can change is the subject of vibe coding and its hidden cost.

Worked example: one table, two features

Butterslip Larder stores a customer's loyalty points on the same record as their delivery address, an old decision with a good reason at the time. Amira asks for a feature: let customers send a gift box to a friend. An AI draft adds a "gift address" by overwriting the delivery address for that order. It works in a demo. In production, the next normal order goes to the friend, and the loyalty points calculation, which reads the address to decide the delivery zone, now awards points at the wrong rate. A developer who knows that table would have added a separate address for the order and left the customer record alone. Nothing in the request mentioned loyalty points. Only knowledge of the codebase did.

A review process that keeps a person responsible

None of this means "do not use AI". It means every change has an owner who understands it. This is the process I would set for a small shop like Amira's.

The rule comes first, in plain words, so the test has something to check against. Passing tests only prove what the tests ask, so a developer still reads every line and must be able to explain it. Anything touching the checkout, payments or customer data gets a second look and a run on a test shop, then ships with a name against it. The responsibility side of this, in writing and in contracts, is in the sibling article.

How much should Amira hand to AI? An expected value worked through

Amira pays her developer by the hour and wants to know, task by task, whether letting AI draft the code saves money once the risk of a mistake is counted. The tool is expected value: weight each outcome by its chance and add them up.

$$E[C] = r \sum_{i} p_i \, h_i + q \, D$$

Here $E[C]$ is the expected cost of doing the task one way, $r$ is the developer's hourly rate, the sum $\sum_i$ runs over the possible outcomes for how long it takes, $p_i$ is the probability of outcome $i$, $h_i$ is the hours in that outcome, $q$ is the probability that a defect reaches customers, and $D$ is what such a defect costs. The probabilities $p_i$ add up to 1. The idea is set out from scratch in Foundations: expected value.

The assumptions are the example's, not measurements. The rate $r = 75$ euro an hour is a round figure. The time outcomes sit inside the range the studies report, with better odds for routine work, where every study finds the largest gains.

Worked example: routine work, the admin seed data

Without AI the job takes 8 hours and carries a 1% chance of a defect that costs €500 to put right:

$$E[C_{\text{manual}}] = 75 \times 8 + 0.01 \times 500 = 600 + 5 = 605$$

Here $75$ is $r$, $8$ is the hours, $0.01$ is $q$ and $500$ is $D$. Expected cost: €605.

With AI drafting and a quick review, Amira assumes a 60% chance it takes 4.8 hours, 30% that it takes 8, and 10% that it takes 9.6, with the defect chance doubled to 2% because review is lighter:

$$E[C_{\text{AI}}] = 75 \times (0.6 \times 4.8 + 0.3 \times 8 + 0.1 \times 9.6) + 0.02 \times 500 = 75 \times 6.24 + 10 = 478$$

Here the hours are the $h_i$ with their probabilities $p_i$, and $0.02 \times 500$ is $q \times D$. AI saves an expected €127 on this job. Hand it over.

Worked example: critical work, the discount change at the checkout

Without AI, the change takes 20 hours, and with the tests above there is a 2% chance a defect still reaches customers, costing €6,000 in lost margin, refunds and a weekend of fixing:

$$E[C_{A}] = 75 \times 20 + 0.02 \times 6{,}000 = 1{,}500 + 120 = 1{,}620$$

Option B lets AI draft with only a light review. Amira assumes a 40% chance of 14 hours, 35% of 20 and 25% of 24, with expected hours $0.4 \times 14 + 0.35 \times 20 + 0.25 \times 24 = 18.6$. But "almost right" code in a checkout raises the defect chance to 6%:

$$E[C_{B}] = 75 \times 18.6 + 0.06 \times 6{,}000 = 1{,}395 + 360 = 1{,}755$$

Here $18.6$ is the expected hours $\sum_i p_i h_i$ and $0.06$ is the raised $q$.

Option C lets AI draft, but follows the full review process above, adding 2 hours to every outcome (expected 20.6 hours) and bringing the defect chance back to 2%:

$$E[C_{C}] = 75 \times 20.6 + 0.02 \times 6{,}000 = 1{,}545 + 120 = 1{,}665$$

Here $20.6$ is the expected hours with the review included and $0.02$ is $q$ restored by the tests and review.

The horizontal axis shows the three ways of making the change. The vertical axis is the expected cost in euro, the developer's time plus the probability weighted cost of a defect. The takeaway: the option with the lowest bill for hours, AI with a light review, is the most expensive once the risk is counted. With a full review, AI and the developer alone come out within €45 of each other, well inside the uncertainty of Amira's guesses. On the checkout, the saving is eaten by the review it needs.

Expected value also hides something. A €6,000 failure the week before Christmas might be one Amira cannot absorb, whatever its probability. When one bad outcome could hurt the business badly, the average is not enough, as I explain in decision theory for founders.

So her answer is a split, not a yes or no: AI drafts the routine, easy to check work, and the critical paths stay with a developer who writes the tests first, with AI as an adviser at most.

Where this leaves the developer

In my experience, ChatGPT and tools like it are assistants, not replacements. They make repetitive tasks faster and are a good sounding board when you are stuck. The evidence agrees on the shape, if not the size: real gains on fresh, well specified work and for people learning, small or uncertain gains for experts in mature code, and a gap between feeling faster and being faster.

What does not move is responsibility. Somebody has to decide what the code should do, write down the rule, catch the case nobody mentioned, and understand the change well enough to stand behind it. Writing code well needs critical thinking, planning, and an understanding of how a change fits into the larger system. Until a tool can do that and answer for it, developers remain essential.

I still prefer to write complex code myself, because I know my own thought process and understand why each part works. AI may assist. The real work is still done by developers who bring the experience, the knowledge and the judgement to the table.

Footnotes

1. Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, arXiv:2302.06590, February 2023, https://arxiv.org/abs/2302.06590

2. Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz, The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers, working paper, February 2025 (published in Management Science), https://economics.mit.edu/sites/default/files/inline-files/draft_copilot_experiments.pdf

3. Joel Becker, Nate Rush, Elizabeth Barnes and David Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, arXiv:2507.09089, July 2025, https://arxiv.org/abs/2507.09089

4. METR, We are Changing our Developer Productivity Experiment Design, 24 February 2026, https://metr.org/blog/2026-02-24-uplift-update/

5. Stack Overflow, 2025 Developer Survey, AI section, 2025, https://survey.stackoverflow.co/2025/ai

6. Stack Overflow, 2026 Developer Survey, AI section, 2026, https://survey.stackoverflow.co/2026/ai