The analysis an expert would run
“Is AI paying off for us?” sounds simple. A good answer isn’t.
Getting the right insights from your data takes expertise: the right metrics, the right denominator, billable rather than seat-covered spend, what to exclude, and how to compare periods. Get one wrong and the answer changes, even when the data doesn’t.
Today we’re launching Milestone Playbooks: expert answers from your engineering data, at the push of a button. Each playbook reads your AI spend, adoption and delivery with Milestone’s know-how in measuring AI in engineering built in. Pick a playbook, set the scope, press Run. The answer comes first and the method sits underneath. Next month it comes out the same way, so you can compare.
Miss one rule, get the wrong answer
Here’s what it takes to get that analysis from a chat assistant:
The rules a good answer needs
Compare Q3 2026 with Q2 2026 for the engineering org. Use billable spend only and show seat-covered usage separately. Exclude bot and automation accounts. Adoption = developers using any AI tool who also committed, divided by active developers. Normalize merged PRs per active developer. Show cost per merged PR beside PR size and cost per 1,000 lines merged. Don’t draw conclusions from small samples. Lead with a one-line verdict, then ranked actions.
Each highlighted rule changes the answer. Count bot accounts and every rate shifts. Count seat costs you’ve already paid and the unit price is wrong. Skip the per-developer view and a bigger team can look like a faster one.
That know-how usually sits with a few specialists, so the answer changes with who asks.
A prompt library gives you someone’s words. A playbook gives you the method.
Know-how in. Answers out.
Milestone’s know-how, packaged into a playbook for each big question. Pick it, run it, compare it over time.
Built in: Milestone’s know-how, built in.
Each playbook packages how Milestone measures AI in engineering: the questions to ask, the data to pull, the metrics and denominators that hold up, and the traps that change a conclusion. It is built from the questions engineering leaders ask Milestone most and the analyses that answered them best.
Setup shows that method before the run: the guardrails and the metric definitions. The report opens with the answer and ends with ranked next steps.
Figure 1. The method, shown before the run: guardrails and metric definitions.
Illustrative data.
Push-button: Pick it. Set it. Run it.
Nine playbooks, one for each big question about AI spend, AI impact, performance and your own usage. Pick one, set the scope and window, press Run. The prior period comes from the window, and the self-check knows who you are.
No analysis to design, no prompt to write. Ask follow-ups in AI Chat, with the report attached, and the report stays as it is.
Figure 2. The gallery: each card names the job and what you set. Illustrative data.
Compare: Same method every time. Compare over time.
Because the method is fixed, a playbook comes out the same way on every run: same playbook, same parameters, same data, same figures, verdict and sections. Every run is kept as it was, so this quarter’s report sits next to last quarter’s.
Figure 3. Last quarter’s briefing, exactly as it ran on Jul 7. Illustrative data.
What every playbook knows
Every playbook is built on Milestone’s know-how in measuring AI in engineering.
Six rules change a conclusion when they’re wrong. Every playbook applies them, and every report says so.
- 01Bots outBot and automation accounts are excluded, and the report says so.
- 02Billable spendCost is billable spend. Seat-covered usage sits on its own “already paid” line.
- 03The right denominatorAdoption uses each team’s own headcount, with numerator and denominator printed.
- 04Size-aware costCost per merged PR sits beside PR size and cost per 1,000 lines merged.
- 05No small-sample verdictsBelow a stated sample size, a comparison is labelled and no conclusion is drawn.
- 06Gaps namedData that isn’t available reads “not tracked”, never zero, and never feeds the verdict.
Nine playbooks at launch
Each one is Milestone’s best-practice analysis for one job, grouped the way the gallery groups them.
AI spend
- MTD AI spend reviewSpend, model efficiency, throughput impact and waste for a single developer, ending in a budget call.
- AI ROI & waste checkCost per PR, tokens not landing in code, and model efficiency against the prior period.
- Agent working reviewHow a developer, team or group works with coding agents: tools, compaction, delegation, model mix and outlier sessions, against fixed thresholds.
AI impact
- Team AI scorecardAdoption, throughput, delivery, cost and quality against the prior equivalent period.
- Leadership AI impact briefingBefore/after or period-over-period impact, normalized per developer, written for a named audience.
- Adoption gaps & non-adoptersAdoption per group on the right denominator, who is not adopting, and whether gaps are closing.
Performance
- Manager performance deep-divePer-member metrics against the BU median, with anomalies, trend and PTO adjustment.
- Cycle-time bottleneck RCAFinds the long tail, classifies where time is lost, and ranks the fixes.
For me
- My AI usage self-checkYour own usage tier, adoption, spend and delivery against your team median.
Which playbooks you see follows your role, the same as AI Chat.
Same data, read the right way
Here’s how it plays out at Corvant, a fictional company with 84 active developers.
Omar Haddad, VP Engineering at Corvant, has the Q3 board meeting on Thursday. He picks the Leadership AI impact briefing, sets the engineering org, Q3 against Q2 and a board audience, and presses Run.
The verdict: “Delivery up, unit cost rising.” Here’s what the playbook reads that a quick look would miss.
Illustrative · Corvant is a fictional company
- Usual readMerged PRs up 19%Playbook readUp 10.9% per developer, because headcount grew from 78 to 84.
- Usual readAI spend up 39%Playbook readCost per merged PR up 16.5%, and not because PRs got smaller.
- Usual read88% adoptionPlaybook read74 of 84 developers, with two groups still below 80%.
- $29The next step. The lead recommendation: hold cost per merged PR at or below $29 through Q4.
One more question before Thursday goes to AI Chat, with the report attached.
Omar asks“Which groups drove the rise in cost per merged PR?”
AI Chat answers“Two groups account for 71% of the increase. Platform engineering moved premium-model share from 31% to 52% of billable spend, and Data & ML ran long agent sessions on two migration projects. Same window, same scope. The report above is untouched.”
Premium means above the org average cost per 1M tokens on paid on-demand usage.
Milestone recommends a target and shows the math. The decision is Omar’s.
Corvant is a fictional company. Every figure in this example is illustrative, not customer data.
The same report next quarter
The right read gets you a good answer once. A fixed method gets you the same read next quarter. A chat answer is rebuilt every time you ask. Ask “How did AI change our delivery last quarter?” on Monday, then again on Thursday:
Monday“Merged PRs are up 19% and cycle time fell 18%. AI is clearly working.”
Thursday“Output per developer rose about 11%, but AI spend rose 39%. The picture is mixed.”
Both are true of the same data: the framing changed, not the facts. Even identical words don’t promise identical output. Thinking Machines Lab ran one prompt 1,000 times at temperature 0 and got back 80 different completions (Thinking Machines Lab, September 10, 2025). And 48% of developers trust AI output only when they can easily verify it (Stack Overflow 2026 Developer Survey, October 6, 2026).
A playbook handles both. Same playbook, same parameters, same data: the same figures, verdict and sections. And every report states its method, so anyone can check it. Here is Omar’s:
How this was produced 7 assumptions · 2 data gaps
- Window
- Q3 2026 (Jul 1 to Sep 30) against Q2 2026, the previous equivalent period.
- Spend basis
- Billable. Seat-covered usage, $18.6K a quarter, on its own line as already paid.
- Exclusions
- 9 bot and automation accounts.
- Denominators
- Adoption = developers using any AI tool who also committed ÷ active developers (74 ÷ 84).
- Normalization
- Merged PRs per active developer (headcount 78 to 84).
- Definitions
- Cycle time = first commit to merge. Post-review change rate = share of merged PRs with commits after the first review.
- Audience
- Board, names withheld.
- Data gaps
- Test-coverage rate is not tracked for this org. Issue data covers 9 of 11 groups connected to Jira.
Last quarter’s briefing, run on July 7, is still in Past runs, unchanged. Next quarter, Omar runs the same playbook and compares.
1.7 → 1.4 dmedian cycle time, Q2 to Q3
7assumptions stated in the report
2data gaps named, not hidden
Get started
Available now in Milestone. If you use Milestone, Playbooks are already in your assistant.
- Open Playbooks in the left nav, marked NEW, or choose “Run a playbook” on the assistant home.
- Set the scope and window. Check the guardrails and definitions, then press Run analysis.
- Read, ask, compare. The answer comes first, the method underneath. Next quarter, run it again and compare.
See a playbook in action.
Book a demo. We’ll run a playbook on a sample org with you, from the gallery to the export.
Or watch the film →