Measuring the effectiveness of Agile product delivery

Jason Yip
3 min readDec 19, 2022

Two main reasons to measure effectiveness

There are two main reasons I see to measure the effectiveness of Agile product delivery:

  1. To not fool yourself… and therefore to learn. Are you actually sure what you’re advocating works?
  2. To build trust and influence. Relying on relationships to be considered effective withdraws from you trust account, showing measurable effectiveness deposits to your trust account.

Effectiveness depends on the product life cycle stage

Graph of product life cycle stages. X-axis time, y-axis revenue. Stage 1: market development with no or low revenue; Stage 2: Growth with rapidly increasing revenue; Stage 3: Maturity with still increasing revenue but slowing; Stage 4: Decline with decreasing revenue
Product life cycle stages

All product capabilities go through a product life cycle and each stage has different goals and therefore different indications of effectiveness:

Market development / innovation

Goal: Discover new market opportunities

Possible indicators of effective product delivery: # of experiments run, hit rate (what percentage of product ideas find traction?), estimated value of new markets discovered

Growth / maturity

Goal: Increase business value

Possible indicators of effective product delivery: # of valuable features delivered, feature development lead time, new user growth, user engagement, user retention, revenue per employee

Decline / commodity

Goal: Reduce ownership / operating costs

Possible indicators of effective product delivery: % of person hours allocated to supporting commodity capabilities, total cost of ownership

Distinguish lagging vs leading indicators

Sometimes what we actually want to measure takes too long to happen and it’s useful to have proxy indicators that indicate interim progress.

Market development / innovation

Goal: Discover new market opportunities

Leading indicators: # of experiments run

Lagging indicators: hit rate, value of new markets discovered

Growth / maturity

Goal: Increase business value

Leading indicators: # of features delivered, feature development lead time

Lagging indicators: new user growth, user engagement, user retention, revenue per employee

Decline / commodity

Goal: Reduce ownership / operating costs

Leading indicators: % of person hours allocated to supporting commodity capabilities

Lagging indicators: total cost of ownership

Quality, Delivery, Cost, Morale

The Lean community tends to talk about several major categories for measuring performance: Safety Quality Delivery Cost Morale

Physical safety issues are generally not as much of a concern with knowledge work, and we can probably bundle psychological safety issues under Morale. Morale is kind of a meta leading indicator. If Morale is trending downward, it’s only a matter of time before it impacts Quality, Delivery, and Cost. Measuring Morale (typically called “engagement” these days) is generally the same across all product life cycle stages.

Market development / innovation

Goal: Discover new market opportunities

Quality: % of experiments that produce useful results

Delivery: # of experiments run, lead time to setup experiments

Cost: actual dollars (e.g., ad dollars reserved for running experiments), # of experiments per employee

Growth / maturity

Goal: Increase business value

Quality: Defect rate

Delivery: # of features delivered, feature development lead time

Cost: revenue per employee

Decline / commodity

Goal: Reduce ownership / operating costs

Quality: MTBF

Delivery: MTTR

Cost: % of person hours allocated to supporting commodity capabilities, servers per employee

“Agile maturity” is more about ideas for improvement than measuring effectiveness

“…the true outcome of a maturity model assessment isn’t what level you are but the list of things you need to work on to improve.”

Martin Fowler, Maturity Model

Agile maturity doesn’t answer “How effective is our product delivery?” but it might answer “What are we actually doing that is leading to this level of effectiveness?”

That is, Agile maturity doesn’t measure product delivery effectiveness, but it might give you ideas for how to improve.

--

--

Jason Yip

Senior Manager Product Engineering at Grainger. Extreme Programming, Agile, Lean guy. Ex-Spotify, ex-ThoughtWorks, ex-CruiseControl