Two main reasons to measure effectiveness
There are two main reasons I see to measure the effectiveness of Agile product delivery:
- To not fool yourself… and therefore to learn. Are you actually sure what you’re advocating works?
- To build trust and influence. Relying on relationships to be considered effective withdraws from you trust account, showing measurable effectiveness deposits to your trust account.
Effectiveness depends on the product life cycle stage
All product capabilities go through a product life cycle and each stage has different goals and therefore different indications of effectiveness:
Market development / innovation
Goal: Discover new market opportunities
Possible indicators of effective product delivery: # of experiments run, hit rate (what percentage of product ideas find traction?), estimated value of new markets discovered
Growth / maturity
Goal: Increase business value
Possible indicators of effective product delivery: # of valuable features delivered, feature development lead time, new user growth, user engagement, user retention, revenue per employee
Decline / commodity
Goal: Reduce ownership / operating costs
Possible indicators of effective product delivery: % of person hours allocated to supporting commodity capabilities, total cost of ownership
Distinguish lagging vs leading indicators
Sometimes what we actually want to measure takes too long to happen and it’s useful to have proxy indicators that indicate interim progress.
Market development / innovation
Goal: Discover new market opportunities
Leading indicators: # of experiments run
Lagging indicators: hit rate, value of new markets discovered
Growth / maturity
Goal: Increase business value
Leading indicators: # of features delivered, feature development lead time
Lagging indicators: new user growth, user engagement, user retention, revenue per employee
Decline / commodity
Goal: Reduce ownership / operating costs
Leading indicators: % of person hours allocated to supporting commodity capabilities
Lagging indicators: total cost of ownership
Quality, Delivery, Cost, Morale
The Lean community tends to talk about several major categories for measuring performance: Safety Quality Delivery Cost Morale
Physical safety issues are generally not as much of a concern with knowledge work, and we can probably bundle psychological safety issues under Morale. Morale is kind of a meta leading indicator. If Morale is trending downward, it’s only a matter of time before it impacts Quality, Delivery, and Cost. Measuring Morale (typically called “engagement” these days) is generally the same across all product life cycle stages.
Market development / innovation
Goal: Discover new market opportunities
Quality: % of experiments that produce useful results
Delivery: # of experiments run, lead time to setup experiments
Cost: actual dollars (e.g., ad dollars reserved for running experiments), # of experiments per employee
Growth / maturity
Goal: Increase business value
Quality: Defect rate
Delivery: # of features delivered, feature development lead time
Cost: revenue per employee
Decline / commodity
Goal: Reduce ownership / operating costs
Quality: MTBF
Delivery: MTTR
Cost: % of person hours allocated to supporting commodity capabilities, servers per employee
“Agile maturity” is more about ideas for improvement than measuring effectiveness
“…the true outcome of a maturity model assessment isn’t what level you are but the list of things you need to work on to improve.”
Martin Fowler, Maturity Model
Agile maturity doesn’t answer “How effective is our product delivery?” but it might answer “What are we actually doing that is leading to this level of effectiveness?”
That is, Agile maturity doesn’t measure product delivery effectiveness, but it might give you ideas for how to improve.
