The optimization tautology

Sprinkled throughout the project portfolio management (PPM) literature and littered among its conferences, one finds the following comparison. An expert selects projects by funding down a ranking and again via optimization. The expert then compares the forecasted value of the two portfolios and declares that optimization is victorious.

This method of evaluating optimization is akin to evaluating a weather forecast by comparing it to other forecasts but never seeing if either forecast is correct. Technically, one cannot evaluate an optimization model by using the model as its own standard of judgment. That's a tautology.

Closely related to the tautology is the mistake of confusing the two uses of the word optimization: as a noun and as an adjective. The practice of optimization (noun) need not produce optimal results (adjective). Optimization produces suboptimal results when it is too sensitive to project evaluation errors or suffers from modeling errors. For examples and explanations, see my discussions:

• What modern portfolio theory reveals about PPM.
• How PPM differs from modern portfolio theory.
• When is optimization suboptimal?
• Where's the feedaback?

I am developing new metrics for evaluating and comparing project selection techniques. One metric applies easily to a wide variety of situations and works with small amounts of data, typical of most PPM. It helps managers compare two selection techniques, such as funding down a ranking vs. optimization, simple optimization models vs. complex ones and PPM project selection recommendations vs. executive overrides.

For organizations with larger portfolios, I've created a metric that offers more incisive information, evaluating the original choice set of projects, the ability of project evaluations to distinguish valuable from poor projects and the trade-off between funding more projects and downstream project success rates. (Funding more projects upstream decreases project success rates downstream.)

Unlike maturity models, which compare processes, my metrics evaluate PPM by analyzing its results.

[an error occurred while processing this directive] [an error occurred while processing this directive]