Why Big Data Could be Messy

The hard side of business—cost, speed, inventory turns, supply chains, capital efficiency—consists of things you can measure with precision. Not surprisingly, the hard side has found a perfect partnership with computing technology, data and analytics. The oldest counting tool is thought to be a Babylonian abacus, used to tally money.

The marriage of the hard side of business and data analytics continues today. Companies such as Walmart in the 1980s, Dell in the 1990s and Amazon and Netflix today are mas- ters of the universe in data analytics.

Meanwhile, what we might call the soft side of business—design, teams, trust, leadership, smarts, story—has existed in its own realm, the world of mystery and intuition. The best practitioners are celebrated as geniuses, not analysts: Steve Jobs at discerning taste, Jack Welch at training leaders, Phil Knight at selling products as stories, Richard Branson at building excitement. These attributes are not easily measured, nor are the skills needed to bring them off easily transferred.

I exaggerate this Mars-Venus divide between the two sides of busi- ness a bit. The best CEOs have always found a way to bridge the divide. Steve Jobs let the hyper-analytical Tim Cook run the hard side of Apple’s operations. Michael Eisner may have rescued Disney, but he did it with the help of Frank Wells, a brilliant COO. Google once was so lopsidedly analytical that it tested 41 shades of blue on its home page for audience response. Today Google gives its graphic designers more leeway—and Google’s look and feel has improved.

Hidden surprises
In this new era of Big Data the question is: Should we leave the soft side in the hands of intuitive geniuses, or should we use Big Data to bring new rigour and rationality to the soft side? Can this even be done?

Big Data, of course, is a wildly overused term these days. I like the explanation given by Viktor Mayer- Schönberger and Kenneth Cukier in their book Big Data: A Revolution That Will Transform How We Live, Work, and Think (Houghton Mifflin Harcourt, 2013). Big Data, they write, is unbounded and unstructured; im- precise but predictive; and can’t show causation, but can show correlation.

In these ways Big Data, with all its messiness, is more like the soft side of business. So can it help design more enchanting products, build crack teams and strong cultures, and make us more adaptable?

This is new territory. Big Data is evolving so fast that it’s not yet clear where it can offer real insight or where it will be a costly distraction. Big Data has had clear wins in fields such as credit card detection. It promises to identify disease from drops of blood. But what can Big Data do for a business leader trying to sell a product or motivate a team?

To see how Big Data is being applied, I spent the summer talking to CEOs, designers, marketers and team builders across industries, to find out where Big Data is proving to be useful.

One answer comes from Tony Fadell, founder and CEO of Nest Labs, a Silicon Valley maker of ‘smart’ thermostats, which save money by observing the way you heat and cool your house. Fadell learned product design at the right hand of Steve Jobs when he helped launch the iPod.

“Does Big Data help Nest design its thermostats?” I asked. “No,” said Fadell. “Great products come from strong points of view. You design them for yourself. You say no to most of the features that data says you’ll need. Steve Jobs was brilliant at saying no. But Big Data shows how people use your product in ways you hadn’t expected. It provides terrific insight into how you might improve the product’s software, communicate with customers and build loyalty.”

That’s just one tip from one CEO. Big Data is a rapidly emerging field of best-use cases, which I’ll be sharing in upcoming columns.

Rich Karlgaard is the publisher at Forbes