Are scoring models valid?
Some project portfolio management (PPM) experts claim that scoring models are logically flawed, offering two possible problems. First, they assert that adding a risk attribute to a payoff attribute (such as NPV) is incorrect; risks and payoffs should be multiplied, not added. Second, they assert an "apples to oranges" problem, claiming that summing qualitatively dissimilar attributes, such as market potential and strategic fit, is an error.
The criticisms are wrong. Like linear regression, scoring models exploit correlations, and correlations resolve both problems. Using the above example, a scoring model does not add market potential to strategic fit. It adds the correlation of market potential with success to the correlation of strategic fit with success. For the same reason, adding a risk attribute and a payoff attribute is logically sound.
Scoring models are valid, and they are especially useful in two situations:
- If one cannot enumerate a project's risks and cash flows, one cannot build good decision trees or capital asset budgeting models but one can still build a scoring model.
- Scoring models admirably limit the propagation of erroneously estimated variables through its calculations. This property prevents a single large, poorly estimated variable from destroying a project's evaluation. In contrast, with decision trees and capital asset budgeting models a large, poorly estimated valuable can destroy project evaluations. (See my discussion, "Revenue forecasting errors dominate decision trees.")
Unfortunately, scoring models have a drawback. Even with the best models, the correlation of project scores with project value may be too low to produce good project selection. My current research on scoring models studies this issue.
One can learn more about scoring models from my discussions including:
- Correlation and the performance of scoring models. (See my discussion "Is my scoring model good enough?")
- Dangers of combining scoring models and optimization (See my discussion, "Scoring Models + Complex Optimization Models = Water and Oil Don't Mix.")