Selection Bias and the Perils of Benchmarking
-------------------------------------------------
By Jerker Denrell Benchmarking has become a trend for companies nowadays. Many company managers and consultants these managers pay for advice rely to the business practices and methods of many different successful companies in making and executing various strategic business decisions in the hopes of experiencing the same fruitful results. The success of these companies can easily blind anyone and anyone can be led by the examples and “benchmark” that these companies set. In most studies, the common grounds of their findings would be that the leaders of these companies are persistent enough to pursue their goals and they were able to persuade others to join them. However, Jerker Denrell stresses in his article that the same attributes hold true to those leaders who failed to meet their objectives. Basically, this has been the whole point of the article – how many different studies neglect those rather unsuccessful companies. The result of which is the classic statistical trap of selection wherein the samples failed to truly represent the whole population. Denrell stated in his article three kinds of traps that company managers fall into when they rely on biased data. The first, which also in his opinion is the most dangerous of all, is the overvaluation of risky business practices. He explained this first trap through graphs he called the “Effects of Bias”. Both graphs depict the relationship between prevalence of risky business practices and company performance. The only difference is that the first one records all companies that have ever implemented the risky practice while the second one excludes companies that failed. The first graph, having included all companies produces a result that on the average, engaging in risky practices reduces performance. The second graph on the other hand results to risky practices and performance having a positive relationship – meaning, the greater the risk the greater positive impact in the company performance is. The author made a good point in this section and he was able to explain very well his stand on the matter. This is also very consistent to what we’ve learned from our finance classes so far which is the risk-and-return trade-off. The second trap is the magnification of current accomplishments by previous achievements and Denrell explained this through a marathon analogy. In his analogy, assuming that the outcome of one race affects subsequent races, winning ten races is less impressive since a victory in the first race gives the runner a higher chance of winning the second, and an even higher chance of winning the third, and so on. The author also made a strong point in this one. If such assumption is correct, it makes so much sense why many companies, despite having bad performance, continue to have high profits. These companies, being the leader in their respective industries will continue to retain, if not improve, their market share. This could be associated to customer inertia and switching costs. The third and last trap is the issue of reverse causality. Many believe that good performance may be rooted the strong culture of the company. However, the author argued that strong company culture could also be brought about the good performance outcome the company has been experiencing. This is parallel to the classic chicken-and-egg riddle of whichever comes first. After discussing these three traps, the author then suggested a way to resolve the problem and that is to get all the data you can on failure. While he is making a clear point on this, sometimes getting all possible data is just a nuisance and costly than it is beneficial. Also he mentioned that it really is impossible to get data that are completely unbiased in which case, economists and statisticians become very helpful for they have developed tools that will correct such selection bias. This article is indeed a very well written one. In the author’s final note, he said that while successful business stories really set out benchmarks for other businesses which for many is the ideal, logic still dictates that managers in pursuit of high performance could attain their goal if they do not just concentrate on these successes but also look on the other side of it – failures – with equal view and bearing.