#32 Are Your Measures Informing or Deforming Your Decisions?

For some, the sound of the word "statistics" coming from someone's mouth impulses them to run far and fast in the opposite direction. Our society seems to be almost proudly fearful of anything to do with mathematics, especially its branch of statistical analysis. "Lies, damn lies and statistics!" they scoff. What a horrible foundation on which to build adequately informed decision-making. Particularly with performance measurement, due to its potent influence on business decision-making, the way performance data is often analysed produces a grotesque parody of useful and usable information.

What can we do about this? One simple step in a good direction is to be aware of how different types of performance data analysis relate to the different types of performance management questions we ask. How does your organisation compare to the following suggestions?

business question 1: Is this performance result getting better (or worse, or not changing)?

For example:

• is the accuracy of our bills improving?

• is cycle time reducing?

• is our cost of inventory stabilising?

• are we getting better at retaining our best staff?

Analyses that can help answer these questions:

• use trend over time graphical methods, such as line charts, run charts and statistical process control charts

• avoid linear trend lines - in the vast majority of cases they explain very little of the pattern of change, and rarely is anything so simple it can be explained by a straight line

• avoid tables comparing this month to last month - two points of data are totally inadequate in accurately evaluating change, due to a natural phenomenon known as variation - you need a good time series of around 20 points

business question 2: What are the main reasons this result is happening?

For example:

• what are the main types of errors on bills?

• what are the 20% of problems that cause 80% of our cycle time blowouts?

• what is the total cost of each of the slowest moving inventory items?

• what are the reasons that staff leave us?

Analyses that can help answer these questions:

• use a bar chart of causal factors or reasons

• even better is to use a Pareto chart (most visually effective when the bars are horizontal, instead of vertical and ordered from largest to smallest)

• avoid tables of numbers as they are much more difficult to interpret than the visual impact of charts

• avoid pie charts, as they are designed to compare a part with it's whole, and encourage misleading visual comparisons between the slices

business question 3: Is this result really related to that result?

For example:

• is the extent of errors on bills related to the workload of our billing staff?

• what is the best lead indicator of cycle time: total throughput, customer ordering lead time or supplier on-time performance?

• if we increase the percentage of inventory items on auto-order to 60%, what percentage change could we expect in inventory costs as a result?

• to what extent is employee turnover related to employee job satisfaction?

Analyses that can help answer these questions:

• use regression analysis to get a quantitative model of the relationship between two measures, also useful to predict changes in one measure as a result of changes in another

• use correlation analysis to get a measure of the strength of the relationship between the two results (called a correlation coefficient)

• more simply, use a scatter plot of the two measures

• sometimes okay to use a line chart of the two measures in time series and visually look for direct or lagged correlations in their patterns over time

• avoid relying on gut feel and hearsay - any relationship needs to be objectively tested

business question 4: How big is this result for us, compared to the same result for them?

For example:

• what is our bill accuracy level like across each of our product streams?

• how does our cycle time compare to our competitors?

• what is the relative cost of each category of inventory we hold?

• is employee retention different between our departments?

Analyses that can help answer these questions:

• use vertical bar charts, with the bars ordered in a way that is logical to the classes or categories you are comparing

• avoid pie charts and tables, for the same reasons listed above

So there are at least two things you could start doing now to ensure that you are analysing your performance data in the most useful way:

1) work out which types of business questions you're asking (or need to ask)

2) choose the types of analyses that are best suited to answering those questions

It's far more worthwhile to use the simplest analysis method well, than to use a more sophisticated analysis method poorly.