The next AI budget review starts with a familiar request:
more capacity for agents.
Usage is growing. Teams are opening more pull requests. AI tools can increasingly show what ran, which models were called, which tools an agent invoked, where it failed, and how much the session consumed.
The CTO has a different set of questions.
Which workflows should get more investment? Where are agents actually giving engineers time back? Where is additional output creating a review bottleneck? Which agents are producing work that survives contact with production? And which have earned enough trust to operate with more autonomy?
Those decisions require more than agent activity.
They require a connection between what AI consumed, what the team actually delivered, how much human effort the work required, and what happened to quality afterward.
When those facts live in separate systems, activity can look like progress long before anyone knows whether engineering improved.
That is the measurement problem emerging as software development becomes increasingly agentic.
Agent telemetry is improving. The investment decision still needs context.
On September 22, GitHub announced OpenTelemetry support in the GitHub Copilot app through enterprise-managed settings.
That is an important step forward for agent observability. Administrators can export agent activity to compatible monitoring systems, inspect model requests and tool use, and investigate execution step by step.
For platform teams, this matters.
If an agent gets stuck in a loop, repeatedly invokes the wrong tool, or follows an unexpected path, the execution trace can help explain what happened.
But an execution trace answers only part of the engineering question.
An agent can complete its run successfully and produce a change the team rejects. Another agent can hit several errors, recover, and produce useful work.
The trace explains how the agent behaved. Engineering leaders still need to know what the behavior produced.
Did someone accept the work? How much did a reviewer rewrite? Did the change reach production? Did it remain stable? And did the workflow ultimately save engineering capacity or simply move work somewhere else?
The next step in agent observability is therefore connecting execution to engineering outcomes.
The missing connection is the work item
A model request has an identifier. A pull request has an identifier. A Jira ticket has an identifier. An incident has an identifier.
If those records cannot be connected, the organization has several views of related activity but no reliable view of the work itself.
The useful unit is the work item: the feature, bug, migration, investigation, or maintenance task the engineering organization intended to complete.
From there, the organization can preserve the relationships around it:
Agent execution→Work item→Code change→Human review→Delivery→Quality
That sounds simple, but the relationship is rarely one-to-one.
An engineer may use several agent sessions to complete one change. One agent run may generate several changes. A reviewer may rewrite much of an implementation while keeping the tests. A task may produce useful analysis without producing any code at all.
That is why simply labeling a merged pull request as “AI-generated” can be misleading.
The question is what AI contributed, what humans contributed, and what the combined workflow achieved.
More agent output can simply move the bottleneck
Consider an illustrative platform team using agents to handle dependency updates.
| Weekly measure | Before agent expansion | After agent expansion |
|---|
| Agent-created PRs | 40 | 80 |
| Accepted and deployed changes | 30 | 30 |
| Active human review and correction | 10 hours | 18 hours |
| AI usage cost | $100 | $160 |
| Changes requiring a fix within 14 days | 2 of 30 | 5 of 30 |
Illustrative figures only. These are hypothetical numbers, not Milestone customer data.
On an activity dashboard, the rollout appears successful. Agent output doubled.
But engineering throughput did not. The team still delivered 30 changes. Review effort increased. AI spend increased. More accepted changes required subsequent correction.
The problem may not be the agent itself.
Perhaps the changes are too large to review efficiently. Perhaps multiple agents are generating duplicate updates. Repository instructions may be incomplete. Reviewers may be repeatedly correcting the same assumptions.
That is exactly where execution traces become valuable: they help investigate the behavior behind an engineering problem. But the engineering problem must first be visible.
Buying additional agent capacity before identifying the constraint could simply create more work for the same bottleneck.
Human-agent alignment becomes visible at the handoff
Agent productivity cannot be measured entirely inside the agent. It becomes visible when the agent’s work reaches a person.
A fast implementation that requires an engineer to rewrite half the change may consume much of the time it appeared to save. A slower agent that produces a smaller change, with the correct tests and context, may create more value because the engineer can confidently accept it.
That suggests a different set of questions:
- Did the output address the task the team intended?
- How much did the engineer have to redirect or rewrite?
- Could the responsible reviewer evaluate it efficiently?
- Did the accepted change remain sound after merge?
- How much human intervention was required to get there?
This also matters for attribution. An agent might produce the first implementation while an engineer supplies the architecture, catches incorrect assumptions, and completes the final change.
Crediting the entire outcome to the agent erases the human contribution that made the work viable.
The goal is to understand whether the combined system is getting better.
Measure the whole workflow, not just the run
Agent economics can become misleading for the same reason. Model or token cost is visible and easy to compare. The actual cost of the workflow is broader.
It can include:
- Successful and unsuccessful agent attempts
- Model and token usage
- Tool or runtime costs
- Active human review
- Corrections and rewrites
- Follow-up engineering work
That leads to a more useful unit of analysis:
Cost per accepted work item = total workflow cost ÷ accepted work items
“Accepted” must have a consistent definition. For a coding workflow, that might mean a change that passes required checks and receives human approval against the original task. For an investigation workflow, it may mean a diagnosis accepted by the responsible engineer.
And acceptance should not be treated as the final outcome. A merged change that needs to be corrected three days later belongs in the quality record too.
The observation window matters. Comparing a six-week-old set of changes with work merged yesterday will distort the result because the newer work has had less opportunity to fail.
The same principle applies to human effort. Elapsed review time is not the same as labor time. A pull request waiting overnight may have a ten-hour cycle time while consuming only twenty minutes of reviewer attention. Both measures can be useful. They answer different questions.
Turn the evidence into four engineering decisions
The purpose of measurement is to make better decisions. For AI-native engineering organizations, the evidence should help leaders answer four increasingly important questions.
| Decision |
Evidence to examine |
Possible action |
| Where should we invest? |
AI spend, accepted output, delivery time, human effort |
Expand workflows producing sustained delivery gains at acceptable total cost |
| Where should we intervene? |
Review queues, active review time, human edits, repeated agent failures |
Improve task scope, context, instructions, or review ownership |
| What does quality permit? |
Acceptance, rework, defects, code retention, follow-up window |
Strengthen checks or narrow the types of changes agents handle |
| How much autonomy should an agent have? |
Owner, permissions, intervention history, observed quality |
Expand, monitor, or restrict agent scope based on evidence |
This is also why organization-wide AI averages have limited value. An agent may perform extremely well on repetitive maintenance and poorly on architectural migrations. A coding tool might create major value for one team and additional review work for another.
The relevant question is: Where, for which work, under which conditions, is AI improving engineering?
Make uncertainty visible
There is another requirement for credible AI measurement: knowing what you do not know.
Different tools expose different levels of telemetry. If one agent provides detailed execution traces and another provides almost none, the first may appear to fail more often simply because its failures are visible.
Similarly, not every agent session can always be reliably attributed to a work item. Those gaps should remain visible. Unknown attribution is better than false precision.
Organizations should understand:
- Which AI tools are covered
- Which repositories and workflows are included
- How many work items have linked agent activity
- Which costs are included
- What human effort can be measured
- How mature the quality observation window is
And they should be careful with causal claims. If AI-assisted work is delivered faster, that shows a relationship. It does not automatically prove that AI caused the improvement. Task difficulty, reviewer availability, team experience, and repository familiarity may all affect the result.
The goal is enough evidence to make better engineering decisions.
How Milestone connects agent activity to engineering impact
Milestone brings these questions into a connected engineering view.
Agent Observability shows how agents contribute to engineering work, including acceptance, retention, and human collaboration.
AI Spend Hub connects AI investment to engineering outcomes, making it possible to examine cost alongside contribution rather than treating usage as value.
Agents Registry provides visibility into the agents operating across the organization.
Agent Governance adds the ownership, controls, and limits required as those agents receive greater autonomy.
Together, those views allow engineering leaders to ask a much more useful set of questions:
- Which teams are converting additional AI investment into delivered work?
- Where is human review absorbing the gain?
- Which workflows are producing sustainable engineering capacity?
- Which agent needs better context or tighter scope?
- Which investments are earning their keep?
- Where has an agent demonstrated enough reliability to do more?
Execution telemetry remains important. It tells you what happened inside the run.
Milestone connects that activity to the engineering environment in which leaders decide where to scale AI, where to intervene, and where to change the investment.
Make your next agent review actionable
Start with one recurring workflow and one decision the team actually needs to make. Name the owner. Define what an accepted result means. Then ask:
- What did AI consume, including unsuccessful attempts?
- What human effort did the workflow require?
- Did the team deliver more comparable work, or did the queue simply move?
- Where did people redirect, rewrite, or correct the agent?
- What happened to accepted work afterward?
- What scope and autonomy does the evidence justify?
- What should change before the next review?
The review should end with an action: expand a workflow, improve its instructions, change the model, reassign review capacity, tighten permissions, reduce agent scope, or stop a low-value workflow.
A dashboard becomes valuable when its evidence changes what the engineering organization does next.
The next phase of agent observability
Better traces make agent behavior easier to inspect. As agents take on more engineering work, leaders need to follow that activity beyond the run: into the work the organization accepts, the human effort required to make it usable, the outcomes it produces, and the quality that remains afterward.
Because the strategic question is whether the engineering organization is becoming better because of AI.
Frequently asked questions
What should a CTO measure before expanding AI agent use?
Look at AI spend, accepted output, delivery time, active human effort, and quality together for comparable work. Increasing agent activity without examining the complete workflow can hide review bottlenecks or downstream quality costs.
What does an agent trace tell you?
An execution trace can show instrumented activity such as model requests, tool calls, retries, timing, and errors. It helps explain how the agent behaved. Engineering outcomes require additional evidence from work management, review, delivery, and quality systems.
How should companies measure human-agent collaboration?
Measure whether the agent addressed the intended work, how much human redirection or rewriting was required, how efficiently reviewers could evaluate the output, and what happened after acceptance.
Should every agent session map to one pull request?
No. A single task may involve several sessions, and one session may contribute to several changes. Measurement should preserve those relationships rather than force a one-to-one attribution model.
When should an agent receive more autonomy?
When evidence within its defined scope supports it: clear ownership, appropriate permissions, manageable human intervention, and consistent quality. Strong performance on low-risk work does not automatically justify autonomy in higher-risk systems.
Where does Milestone fit?
Milestone connects AI spend, agent activity, engineering output, human collaboration, quality, and governance so engineering leaders can decide where to expand AI, where to intervene, and how agent scope should evolve.