more-arw search

Most Metrics Are RELATIVE, Not Absolute

What are metrics for? When properly used, most are nothing more than a relative tool for comparing one time period – or company, or country, or person – to another, or actual results to planned or expected results. They are not an absolute score, meaningful on its own. It’s important to remember this, whether the metric involves business, government, sports, or national economic performance.

I saw yet another article about a change in the U.S. unemployment rate (downward this time) with the same tired old qualifying observation that the change doesn’t factor in people who have given up and chosen to stop looking for work, thereby understating the true unemployment rate by reducing the number of verifiably, measurably unemployed.

To this news, my reaction is, “So what?” Let’s call this phenomenon “the flaw,” and ask ourselves:

  • Is the “flaw” a truly material number? (A home exercise: compute how many unemployed have to leave the labor force to move the unemployment rate from 7% to 6%.)
  • Is the “flaw” any more or less pronounced than it was the last time the rate went up (or down) by a similar amount?
  • Are the causes of the “flaw” uniformly negative – i.e., people leaving the workforce specifically because they’ve given up – or are there also positive explanations? For example, has someone else in the household who was unemployed now found a job? Have some people found “off the books” employment?

And most important, is the “flaw” so severe that that you can’t really tell whether the problem of unemployment is getting better or worse? If so, then the metric should be thrown out and replaced with a better one. If not, the hand-wringing is pointless.

Many, many metrics have similar flaws, but that doesn’t mean that they aren’t immensely useful even so.

“Painting with Numbers” is my effort to get people to focus on making numbers understandable.  I welcome your feedback and your favorite examples.  Follow me on twitter at @RandallBolten.

Comments

Topic Expert
Wayne Spivak
Title: President & CFO
Company: SBAConsulting.com
LinkedIn Profile
(President & CFO, SBAConsulting.com) |

What if the "flaw" (like the ubiquitous "the poll has an error rate of +/- 1%") could be quantified.

Is it still a flaw?

By the way in your example, our government could never determine the flaw, so yes that figure is somewhat useless, especially to those in the market and still not finding work...

Randall Bolten
Title: CEO
Company: Lucidity
LinkedIn Profile
(CEO, Lucidity) |

If the "flaw" can be quantified, then it may be less of a "flaw." For example, your "error rate of +/-X%" can help answer my first question, namely, whether the error is material to useful understanding of the metric.

Randall Bolten
Title: CEO
Company: Lucidity
LinkedIn Profile
(CEO, Lucidity) |

Regarding your observation that the metric is useless to those in the market and not finding work: yes, that's true. A metric like unemployment rate is almost never meaningful to a specific individual's situation, but that isn't the purpose of that metric. The fact that a disease has a 94% survival rate is of no consolation to the 6% who don't survive -- but that is the kind of obvious and pointless observation that politicians in particular make over and over and over.

Topic Expert
Wayne Spivak
Title: President & CFO
Company: SBAConsulting.com
LinkedIn Profile
(President & CFO, SBAConsulting.com) |

Sounds like it harkens to the old adage:

There are lies, damm lies and statistics; and 93.4% of those are made up.

Topic Expert
Regis Quirin
Title: Director of Finance
Company: Gibney Anthony & Flaherty LLP
LinkedIn Profile
(Director of Finance, Gibney Anthony & Flaherty LLP) |

The statistic has a value when trending. People say the statistic is not accurate, as it does not capture those individuals that left the workforce. But we cannot quantify that number. Is it 2 or 2 million?

Statistics are only indicators. Even financial statistics are indicators. A P&L is never accurate until after the audit, and at that point the degree of accuracy is only about 95%. You will only have 100% accuracy after the company closes and you can fully count the widgets left and close the doors.

But that does not mean we should stop using these indicators - leading or lagging.

In 2006, the December unemployment rate was 4.4 as reported by the BLS. But in December 2009 the unemployment rate was 9.9 (BLS). I can say personally I felt better and more confident in 2006, than I did in 2009. Any stand-alone statistic can be questioned and should be questioned.

But absent playing with the measure components, a statistical trend has a high degree of validity.

Randall Bolten
Title: CEO
Company: Lucidity
LinkedIn Profile
(CEO, Lucidity) |

Excellent points, Regis.

Topics: