Design AI Productivity Metrics That Survive Goodhart’s Law
Suppose leadership rewards teams for increasing the percentage of “AI-assisted pull requests.” The dashboard rises. Did productivity improve, or did people learn which box to tick? Before launching that metric, I would run a consequence-mapping session:
| Intended behavior | Plausible adaptation | Counter-metric |
|---|---|---|
| try useful assistance | label trivial PRs as assisted | retained task outcome |
| ship faster | split work into tiny PRs | lead time per task |
| share adoption | avoid difficult non-AI work | task-mix distribution |
| accept suggestions | reduce review scrutiny | rollback and defect rate |
The Metric Card
The metric card should make disagreement possible:
- name: ai_assisted_pr_share
- purpose: detect workflow adoption, not productivity
- owner: developer-experience
- known_game: self-label inflation
- counter_metrics: [task_mix, review_minutes, rollback_rate]
- review_date: 2026-08-19
- retire_when: classification cannot be audited
Evaluating the Metric
Then interview both high and low scorers without treating the score as performance. Ask what work disappeared, what new verification appeared, and what behavior the dashboard encouraged. Include an anonymous channel: a metric cannot reveal pressure if challenging it carries career risk.
The SPACE framework argues that developer productivity cannot be captured by one dimension. That is especially relevant when AI telemetry is easy to count but verification and rework are harder to observe.
My launch gate is not “the metric is accurate.” It is: teams can inspect its definition, challenge its interpretation, and show where it changes behavior. If the counter-metrics diverge, pause incentives before refining the chart.
What behavior would your current AI dashboard accidentally reward?
Top comments (0)
Comments
No comments yet. Start the discussion.