Study: Huge success in business is due to luck

Bestselling business books promise to teach you the winning formula and reveal the secrets of success. But the inconvenient truth is that exceptional successes in business are largely based on luck.

No rule exists for achieving exceptional performance because it usually requires doing something different or novel and there can be no recipe for such innovation.

My new research provides systematic evidence that luck plays a critical role in such performance, not only in business but also in music, movies, science and professional sports.

A key finding is that more can be gained by paying more attention to “second best”. Let’s look at the music industry. If a new band or musician has a top-20 hit, should a music label immediately try to sign them?

My analysis of 8,297 acts in the US Billboard 100 from 1980 to 2008 would suggest not. Music label bosses should instead be looking to sign up those reaching positions between 22 and 30, the “second best” in the charts.

A common feature of many artists charting in the top ranks is that they enjoyed a “runaway success”. A classic example is Gangnam Style by Korean artist PSY. The music video went viral beyond anyone’s foresight.

Since such an outcome involved exceptional luck, PSY’s success is unsustainable. Artists charting in the top 20 will likely see their next single achieve between 40 and 45 on average, regressing disproportionally more to the mean than their lower-performing counterparts.

Those charting between 22 and 30, meanwhile, have the highest predicted future rank for their next single.

Their less exceptional performances suggest that their successes depend less on luck, making their performances a more reliable predictor of their merit as well as future performances. This is where music label bosses will find the hidden gems.

The same happens in the business world. For example, the fastest-growing firms – such as those on Fortune’s 100 Fastest Growing Companies list – usually attract the most media attention, investment and imitation.

My results show that the consecutive-year growth rates of firms are almost random but a systematic “less-is-more effect” can occur.

So businesses with the top current growth rate (more than 34 per cent per annum) have a significantly lower expected growth rate next year than firms with a high but less extreme current growth rate (between 32 per cent and 34 per cent per annum).

This suggests that top performers are not only luckier than the rest but also become predictably worse.

However, selling the problematic idea of learning from the most successful continues to prosper.

For example, many business bestsellers, such as In Search of Excellence, the most widely owned book in the US between 1986 and 2006, share a formula. First, select a few successful firms that beat the odds and achieve excellence.

Then analyse the shared practices of these firms from when they moved from “good to great” and frame these practices as the principles for others that aspire to become great.

An overlooked caveat is that the exceptional performances featured in these bestsellers typically do not last.

Take the 50 firms featured in the three most popular business bestsellers: In Search of Excellence, Good to Great and Built to Last.

My research shows that the significant improvements of these firms (good to great) before being featured were followed by systematic disappointments.

Of the 50 companies, 16 failed within five years after the books were published and 23 became mediocre as they under-performed in the S&P 500 index (which represents the average performance expectation of the 500 largest public companies in the US).

Only five out of the remaining 11 firms maintained a similar level of excellence compared to when they were featured in the books.

What happened after becoming great is not enduring greatness but, instead, strong regression to the average. Nevertheless, such a misleading “success formula” is still popular in business media and education. There is also a growing number of lists featuring top-performing firms, CEOs and entrepreneurs due to uncertainty.