The Metrics That Stopped Working
Last month, a senior engineer on my team shipped a complete authentication system in two days. The code was clean, well-tested, and integrated seamlessly with our existing infrastructure. Five years ago, this would have taken two weeks. The difference? She spent those two days architecting the solution and directing Claude and GitHub Copilot to generate the implementation.
Her velocity metric looked incredible. Her lines-of-code count was through the roof. But here's the uncomfortable question: did those traditional metrics actually capture what made her valuable to the organization?
They didn't. And if you're still using them to evaluate your engineering team, you're measuring the wrong things.
What AI Actually Changed
The shift isn't subtle. When AI handles implementation details, the engineer's role fundamentally transforms. Your developers are becoming orchestrators rather than typists, architects rather than bricklayers.
Consider a typical feature development cycle before widespread AI code generation. A developer would spend perhaps 20% of their time thinking through the architecture, 60% writing implementation code, and 20% debugging and refining. The visible output—lines of code, pull requests, commits—heavily weighted toward that middle 60%.
Now? That same developer might spend 50% of their time on architectural decisions, 20% prompting and directing AI tools, and 30% reviewing, integrating, and refining the generated code. The thinking-to-typing ratio has inverted, but your metrics probably haven't caught up.
The Architecture-First Mindset
When implementation becomes commoditized, architecture becomes everything. But what does "good architecture" actually mean in this context?
It means your developer understands the second-order consequences of technical decisions. When that engineer built the authentication system, she didn't just think about login flows. She considered:
- How the auth system would interact with the existing microservices
- What the migration path looked like for legacy users
- Which security standards applied and why
- How the system would scale as the user base grew
- What the testing strategy should be across integration points
AI tools can generate an OAuth implementation. They can't decide whether OAuth is the right choice for your specific context, user base, and security requirements.
This is the skill that separates senior engineers from junior ones in the AI age—and it's nearly invisible to traditional metrics.
Measuring What Matters Now
So what should you measure instead? The answer requires looking at outcomes and decisions rather than output and activity.
System Design Quality: Track how often architectural decisions need to be revisited or reversed. When an engineer designs a system, how long does that design remain valid? Do subsequent features build cleanly on top of their work, or require constant refactoring?
One team I advise implemented a simple practice: three months after shipping a major feature, the tech lead reviews whether the architectural decisions held up. Did the abstractions prove useful? Were the integration points well-chosen? This backward-looking assessment reveals far more about engineering judgment than commit frequency ever could.
Decision Documentation: The best engineers in an AI-augmented workflow leave a clear trail of why. When AI generates the what, the human-contributed why becomes your most valuable artifact.
Start measuring documentation quality and completeness. Not the auto-generated API docs—those are commodity now. I mean the architectural decision records, the design docs that explain trade-offs, the comments that clarify business logic that AI can't infer from context.
Cross-System Thinking: How effectively do engineers navigate complexity across system boundaries? When building a feature, do they identify all the systems it touches? Do they spot potential conflicts or opportunities for consolidation?
This manifests in fewer production surprises and smoother deployments. You can measure it through incident retrospectives: how often are issues traced back to unconsidered system interactions versus implementation bugs?
The Code Review Transformation
Code review has fundamentally changed. You're no longer primarily checking for syntax errors or basic logic flaws—AI is pretty good at avoiding those. Instead, reviews should focus on:
- Whether the generated code actually solves the right problem
- If the architectural approach is sound
- Whether the code fits into the broader system context
- If there are security or performance implications the AI missed
A useful metric: track what percentage of code review comments address architectural or contextual issues versus implementation details. A healthy ratio in an AI-assisted environment might be 70/30 or 80/20 favoring higher-level concerns. If your reviews are still mostly about code style and basic bugs, either your team isn't effectively using AI tools, or they're not thinking architecturally enough.
Prompting as a Core Skill
There's a new technical skill that doesn't fit into traditional engineering metrics: the ability to effectively direct AI code generation. This isn't about knowing magic prompt words—it's about clearly decomposing problems and communicating requirements.
Engineers who excel at this share common traits. They break down problems into well-scoped chunks. They provide relevant context without overwhelming the AI. They recognize when generated code is subtly wrong versus obviously broken.
You can assess this through pair programming sessions or by reviewing how engineers interact with AI tools. Do they generate working code on the first or second attempt, or do they spend hours fighting with the AI? Do they blindly accept generated code, or do they critically evaluate it?
The Danger of Proxy Metrics
Here's what not to do: don't just add "AI-assisted features shipped" to your existing dashboard and call it done. That's still measuring output, not impact.
I've seen teams fall into the trap of celebrating velocity without questioning value. Yes, features ship faster with AI assistance. But are they the right features? Do they solve real problems? Are they built on solid architectural foundations?
One company I worked with celebrated a 3x increase in feature velocity after adopting AI coding tools. Six months later, they were drowning in technical debt from poorly-considered architectural decisions made in the rush to ship. The metrics looked great. The codebase was a mess.
Building the Evaluation Framework
Start with impact, then work backward to measurable indicators. What outcomes matter for your business? System reliability? Feature adoption? Developer satisfaction? Time-to-market for new products?
For each outcome, identify the engineering decisions that most influence it. Then create feedback loops that help engineers see the consequences of their architectural choices.
This might mean:
- Regular architecture review sessions where teams present and critique designs
- Structured retrospectives that explicitly examine architectural decisions
- Mentorship pairings focused on system design thinking
- Career ladders that emphasize architectural judgment over code volume
The Human Advantage
The engineers who thrive in this new environment aren't necessarily the fastest coders—they never were. They're the ones who understand systems, who see connections, who make thoughtful trade-offs.
They're the ones who know when to use AI to accelerate implementation and when to slow down and think deeply. Who recognize that generating code quickly is only valuable if it's the right code, solving the right problem, in the right way.
Your metrics should reflect this reality. Measure judgment, not just output. Evaluate architectural thinking, not just implementation speed. Recognize that in the age of AI code generation, the most valuable engineering skill is knowing what to build and how it should fit together—not typing it out character by character.
The best developer on your team might soon be an AI. But the best engineer will always be the human who knows what to ask it to build.