The renewal question arrives before the evidence
Within the next two quarters most engineering organizations will be asked a question they cannot currently answer with data: which of our AI coding tools do we keep?
The question usually arrives from finance, and it arrives reasonably. AI line items that were rounding errors in 2024 now sit beside cloud and headcount in the budget review. Several tools were adopted in parallel, some through procurement and some through a team lead’s corporate card. Contracts are coming up. The obvious move is to pick one, standardize, and take the saving.
The obvious move is also the one most likely to be wrong, because the evidence available at the moment of decision is almost always the wrong evidence. Spend per tool, seats assigned, and licence utilization describe the meter. None of them describes what the tool returned. A consolidation decided on those three numbers is a decision about price made without any measure of what the spend produced, which is another way of saying it is a guess with a spreadsheet attached.
There is a comparison that holds up. It requires cutting the portfolio a different way.
The portfolio moved faster than the contract cycle
Start with how unstable the ground is. JetBrains’ Developer Ecosystem Survey, fielded with more than 15,000 professional developers between May and July 2026, found 90 percent using AI coding agents at work at least weekly and 68 percent using them daily.1 That much is expected. The interesting part is the movement underneath the total.
In JetBrains’ January 2026 pulse, GitHub Copilot led work adoption at 29 percent, with Cursor and Claude Code tied at 18 percent.2 Seven months later, Claude Code was at 39 percent globally and 47 percent in the United States, Copilot had fallen to 21 percent, Cursor to 12 percent, and Codex had gone from 3 percent to 16 percent.1 The leader changed, two incumbents declined, and a tool that was a rounding error became a top-three choice, inside a single two-quarter window.
Reported work adoption, JetBrains AI Pulse (January 2026, 10,000+ developers) and Developer Ecosystem Survey 2026 (May–July 2026, 15,000+ developers). Percentages are per tool and are not mutually exclusive.
Two things follow. First, the shares sum to well above 100 percent, which is only possible if a substantial number of developers work with more than one tool. A portfolio is not an accident of governance in most organizations. It is what practising engineers assembled.
Second, and more awkward for procurement, a renewal negotiated on 2025 usage is a decision about a portfolio that no longer exists. The contract cycle is annual or longer. The tool mix is revising itself roughly every two quarters. Any process that samples the evidence once, at signature, is sampling something stale.
Why per-tool totals cannot answer a portfolio question
The deeper problem is not staleness. It is that per-tool totals measure the wrong object.
Look at what the research says about where AI-assisted work helps. DORA’s ROI of AI-assisted Software Development report, published in May 2026, cites Stanford research putting productivity gains around 35 to 40 percent on simple greenfield tasks and at 10 percent or less on complex legacy code.3 METR’s randomized trial of 16 experienced open-source maintainers working on their own repositories, averaging more than a million lines of code, measured those developers as 19 percent slower with AI tools available, while the same developers believed afterwards they had been 20 percent faster.4
The effect of workload is large enough that a tool-level average can conceal the comparison you actually need to make. Consolidating on price compares the tools without comparing the work.
That is the whole argument in one line. A tool that is excellent at scaffolding a new service and mediocre inside a twenty-year-old billing system is not “a worse tool” than one with the reverse profile. It is a different instrument. Comparing their invoices is comparing the price of two things that were doing different jobs.
Self-reported comparisons will not rescue the decision either. METR’s February 2026 survey of 349 technical workers found a median self-reported speed gain of roughly 3x against value gains of 1.4 to 2x, and noted that in its earlier experiments respondents overestimated AI’s effect on their own task time by around 40 percentage points.5 METR has since redesigned its experiment, in part because 30 to 50 percent of developers declined to submit tasks they did not want to attempt without AI.6 If a research organization with a controlled protocol finds the measurement this hard, a survey of team leads two weeks before a renewal is not evidence. It is sentiment with a deadline.
The workload cut
The fix is to stop comparing tools against each other and start comparing each tool against the work it was pointed at. We call this the workload cut. For renewal analysis, we’ve found it useful to separate engineering work into four broad workload classes that behave differently under AI assistance.
CLASS 01
Greenfield and scaffolding
New services, endpoints, test harnesses, internal tools, migrations of format rather than meaning. Few constraints from existing code. This is where the largest measured gains appear, and where nearly every vendor benchmark is set.
CLASS 02
Bounded change in mature code
A defined change inside a system with existing behaviour, tests, and consumers. Most enterprise delivery lives here. Gains are real but modest, and review effort is the swing factor.
CLASS 03
Cross-cutting migration and refactor
Framework upgrades, dependency sweeps, API deprecations across many repositories. High volume, high blast radius, and the class where agent runs and retry behaviour dominate cost.
CLASS 04
Investigation and incident work
Diagnosis, root cause, reproduction, and the fix that follows. What moves is time to resolution and repeat incident rate, not merged lines.
The classes are not a maturity ladder and they are not exclusive. They exist because each one answers the renewal question differently. A tool can be indispensable in class 01 and actively expensive in class 02, and a portfolio-level average will hide both facts.
Figure 1. The workload and tool fit matrix. The value of the grid is the blank cells: they show exactly which comparison a renewal is currently being made without.
What to measure inside each class
Once work is cut by class, the measures are the same in each one, which is what makes the comparison legitimate.
Cost per accepted change. Total cost of the attempt, including retries and abandoned sessions, divided by changes that were accepted in review and merged. We set out why this denominator beats cost per token in High token usage is not waste.
Cost per durable change. The same ratio restricted to changes still standing after the observation window, which is the only version that survives contact with post-merge reality. The full method is in Your AI code passed review, production sent the bill.
Review effort per AI-assisted pull request. Reviewer time, comment density, and review cycles. A tool that halves authoring time and doubles review time has not saved anything, it has moved a cost onto a scarcer person. This is the mechanism behind the review bottleneck.
First-time merge rate. The share of AI-assisted pull requests merged without a rework cycle. It is the cheapest early signal of fit, and it moves before cost per durable change does.
Seat utilization against class. Not just who logged in, but which classes of work each seat was used on. Premium seats assigned to people doing only class 01 work are the most recoverable spend in most portfolios.
Why this matters
The denominator decides the answer
Comparing tools on cost per seat rewards whichever vendor priced lowest. Comparing them on cost per accepted change inside a workload class rewards whichever tool did the work the organization actually has. Those two comparisons frequently disagree, and only one of them is about engineering.
One cost note worth carrying into the model. Variable pricing does not average out the way seat pricing does. Anthropic’s own guidance for Claude Code puts enterprise usage at roughly $13 per developer per active day and $150 to $250 per developer per month, with 90 percent of users staying under $30 on an active day, and it attributes the variance to model selection, codebase size, and usage patterns such as running multiple instances.7 The same documentation notes that agent teams can use approximately seven times the tokens of a standard session.7 A class 03 migration run by parallel agents and a class 01 scaffolding task are not comparable line items even within one tool, which is another reason the workload cut has to come first.
Four decisions the evidence can actually support
Cut the work, measure the classes, and the portfolio question resolves into four decisions rather than one. They are not the same decision and they do not need the same confidence.
DECISION 01
Consolidate
Where two tools produce indistinguishable results inside a class, the overlap is duplication. Cut it, and reassign the seats rather than banking the whole line.
DECISION 02
Keep the overlap
Where one tool is measurably better on a class that carries real delivery weight, the overlap is not waste. Price it against what its removal would add to review and rework.
DECISION 03
Re-scope the seat
Where the gap is in how a tool is used rather than what it is, change the assignment, the model guidance, or the workflow before changing the contract.
DECISION 04
Renew short
Where the data does not exist, buy time instead of certainty. A short extension is cheaper than a multi-year commitment to the wrong portfolio.
The fourth is the one most often skipped and most often correct. A short renewal carries a premium, usually a modest one. A three-year commitment made on seat counts carries the difference between the right portfolio and whichever one happened to be cheapest at signature, compounded over three years, in a market where the leader changed twice in eighteen months.
If you are heading into a renewal without class-level evidence, the first question is not which tool to cut. It is whether your delivery data can currently distinguish these four classes at all. See how Milestone segments it.
The evidence window
Evidence for a renewal has a shelf life at both ends. Too short and it is noise. Too long and it describes models and prices that have been superseded. As a practical starting point, use an instrumented quarter: long enough for accepted output, review effort, and early rework to separate across classes, short enough to still describe the current portfolio.
What has to be in place on day one of that window:
- Work items classified into the four classes, applied consistently, ideally derived from repository and issue signals rather than a manual tagging exercise nobody will sustain.
- Tool and model attribution on each change, at the level of the contributor and the run, not the monthly invoice.
- Review and rework signals joined to the same changes, so authoring cost and downstream cost sit in one record.
- Agent runs included, with retries and abandoned sessions counted against the workload they were attempting.
- Seat assignment visible against class, so unused and misassigned seats surface before the negotiation rather than after it.
If a renewal is due sooner than that, the window has already closed. That is the argument for decision 04, and it is a better argument than any leverage a procurement team will find in the same period.
How Milestone supports a portfolio decision
Milestone was built for the join that this decision needs: AI cost on one side, engineering signals on the other, connected at the level of the change rather than the invoice.
Includes an AI stack audit that benchmarks tools such as GitHub Copilot, Cursor, and Claude Code against your own baseline on review cycles, pull request size, and first-time merge rates. This is the tool-versus-tool comparison a renewal needs, measured on your codebase rather than a vendor's.
Brings engineer time, tool usage, model cost, review effort, and rework into a cost per feature, with collaboration modes showing whether a change was human-written, AI-assisted, AI-generated, or reworked. That distinction is what turns a class-level comparison into something defensible.
Exposes what agent runs actually did, including acceptance and retention rates and the pull request lifecycle from contribution through human edits to merge. Retries and abandoned runs are the cost class 03 work hides best.
Unifies tool, licence, seat, token, model, and agent cost into one ledger, and flags underused seats by team with a reclaimable total. It is the side of the equation procurement already asks for, in a form that can be joined to the other three.
None of this removes judgement from the decision. It removes the part of the decision that was being made on the only numbers that happened to be available.
A renewal checklist for engineering leaders
Six questions. Each one tests whether a consolidation decision would be evidence or preference.
Can we separate our delivery into workload classes without a manual tagging exercise?
If classification depends on people remembering to label work, the evidence will not survive the quarter.
Do we know cost per accepted change by tool and by class?
By tool alone is a budget view. The class dimension is what makes it a decision.
Is review effort on AI-assisted changes counted as a cost of the tool?
If it is not, every tool that moves work into review will read as a saving.
Can we see which seats were used on which classes of work?
This is where the reclaimable spend is, and it is usually recoverable without removing anyone's access.
Are agent retries and abandoned runs attributed to the workload that triggered them?
Unattributed retry cost looks like tool cost, and the wrong tool gets cut.
Would our answer change if the vendor's benchmark were excluded?
If not, the decision is being made on someone else's workload.
Six negatives does not mean the portfolio is wrong. It means that if it is wrong, nobody currently has a way to tell.
Conclusion
Consolidation is a legitimate move. Portfolios do accumulate duplication, teams do hold seats they have stopped opening, and there is real money in tidying that up. The mistake is treating consolidation as a pricing exercise when the variable that decides its outcome is fit.
The organizations that come out of this renewal cycle ahead will not necessarily be running fewer tools. They will be the ones who can say, per class of work, what each tool returned for what it cost, and who therefore know which overlaps were duplication and which were the portfolio doing its job.
Milestone AI Spend Intelligence
Know which AI tools to keep before the contract decides for you.
Connect tool, token, and agent cost with accepted output, review effort, rework, and delivery, segmented by the work your teams actually do.
FAQs
1. What is AI coding tool consolidation?
It is the decision to reduce the number of AI coding assistants and agents an engineering organization pays for, usually taken at renewal. Done well it removes duplicate spend without removing capability. Done on price alone it removes capability that was not duplicated.
2. Why is spend per tool a poor basis for a consolidation decision?
Spend per tool answers what each tool cost. A portfolio decision needs to know what each tool returned, and return varies more by the kind of work than by the tool. Two tools with identical invoices can perform very differently once the work is separated into greenfield, bounded change, migration, and investigation.
3. How long does it take to gather evidence for an AI tool renewal?
As a practical starting point, plan for an instrumented quarter before the decision. That is long enough for accepted output, review effort, and early rework to separate across workload classes, and short enough that the result still describes current models and pricing.
4. Should engineering standardize on a single AI coding tool?
Standardize where measured fit is indistinguishable across tools, because that overlap is real duplication. Keep the overlap where one tool is measurably better on a workload class that carries delivery weight, and price that overlap against what replacing it would add to review and rework.
5. What metrics should an AI tool renewal be based on?
Cost per accepted change and cost per durable change, segmented by workload class, alongside review effort per AI-assisted pull request, first-time merge rate, rework and post-release fixes, and seat utilization by class. Totals per tool are useful for budgeting and close to useless for choosing.
6. What should we do if the data does not exist yet?
Renew short rather than deep. A short extension carries a modest premium and buys an instrumented quarter. A multi-year commitment made without evidence carries the difference between the right portfolio and the one that happened to be cheapest at signature.