I have to admit that I’m sick of all this hype about big data and how it is the answer to all our problems. As I’ve said again and again, there’s no such thing as big data in business. Relative to our ability to process it, data has always been big. And, in business, big has always been meaningless. Furthermore, in business, we’ve always been able to process as much data as we need to in reasonable amounts of time if we made good technology decisions.
And I’m even sicker of the fact that some people think we can replace science with math and processes with computer programs. We never could, and for the foreseeable future, where AI (artificial intelligence) will not be a reality, we can’t. Thinking like this is what causes economists to latch onto, and promote, financial policies that, seem good in theory but, in practice, result in economic collapse when taken to extremes.
The reality is that science can never be replaced by math and automated prediction. Not only is the author of this HBR blog post on why data will never replace thinking right when he says that it’s only by trying to come up with our stories (hypothesis) beforehand, then testing them, that we can reliably learn the lessons of our experiences — and our data, but it’s only by coming up with hypothesis, and putting plans into actions that we can beat the competition and gain market share in the global market. Look at the giants of industry today. Did Apple become the dominant first in the e-Music industry by letting Microsoft, Sony, Samsung, etc. develop their music players and music stores first, analyzing customer responses, and then introducing their offering? Or did they become the dominant force by using their brains to try and figure out what the market, and customers, were missing, using the best creative and engineering talent to design a solution, and then releasing that product on the market? It was the latter solution — the solution that required big brains that won the market. Similarly, Walmart became the biggest retailer not by asking consumers want they wanted, but by predicting what the average consumer really wanted — a one-stop department store that met most of their basic needs at low prices with a consistent product and service offering across each store for the mobile consumer.
This isn’t to say that data isn’t important, it is, just that it won’t solve all your problems and that, beyond a certain point, more data doesn’t help. Remember, statistically speaking, you only need 384 data points to have 95% confidence with a confidence interval of 5 on a population of 1,000,000. If you want a confidence interval of 3, you only need 1,066 data points, and if you want a confidence interval of 1, you only need 9,513. Beyond a certain point, more data doesn’t add much confidence and the only way you’re going to get more insight is to see it inside your head.
So keep your big data. I’ll use my brain instead. How about you?