… then trust fully and completely as the one thing that Big Data can do (besides sucking up a lot of your cash for a dubious ROI), when properly mined, is overcome the Three Cognitive Traps that Stifle Global Innovation.

How? We’ll get to that, but first let’s explain what the three cognitive traps are.

The Experience Bias

Referred to as the availability trap by the authors of the HBR post, it refers to the fact that many people assume an element of a culture that they, and their peers, are familiar with is representative of that culture. The example given by the authors is that Brits see Chicken Tikka Masala as representative of typical indian cuisine as that is what is common in the British curry houses. Similarly, most North Americans think that fried rice and egg rolls are representative of typical Chinese cuisine, as that is what comes with just about every combination plate in every Chinese restaurant. In both cases, this is not true.

Similarly, in the business world, most executives in the developed world believe that the urban, affluent, rapidly growing middle class is representative of the population of a market in a developing country they are going after. The urban middle class is only a small percentage of the population in many developing countries, and not representative of the market as a whole.

The Confirmation Bias

The confirmation bias is when we use ambiguous data as clinching evidence of our hypothesis. The example given by the authors was how a European multinational, despite being told by their Indian salesforce that their building material product line was over-engineered for much of the Indian market, refused to listen (believing that the local salespeople just did not have the required skills to sell the product) and sustained years of losses, including two country CEOs, before it saw the error of its ways. Why did this happen? Early on, a global product executive visited India and happened to be present when a single, stellar, salesperson encountered a high-value customer, patiently demonstrated each and every superior product feature, and, after a significant amount of time, finally made the sale. This one data point of success was focussed on despite the fact that there were countless data points of failure.

The Variance Bias

The variance bias (known to psychologists as out-group homogeneity bias) is where we (drastically) underestimate variance in distant cultures, grouping all Chinese consumers into one market segment, for example, while separating New Yorkers and San Franciscans into two completely different market segments (due to our familiarity with both market groups and in-depth knowledge about their differences relative to our understanding of the Chinese marketplace). (There are 1.35 Billion people in China. Do you really think they are all the same? There should be considerably more market segments in China than in the US.)

With enough data, and the willingness to blindly trust the data, Big Data will overcome all three of these biases. Since big data relies on transactions and facts, and not the limited pool of experience associated with a decision maker, the real patterns (and not the perceived one) will quickly emerge. Similarly, outliers (such as the example of the stellar salesperson who got lucky) will quickly be eliminated and bad hypotheses will not be confirmed. Finally, a good algorithm won’t group data that doesn’t belong together and if it takes 25 clusters to properly segregate the data, the algorithm will return 25 distinct clusters for human analysis and interpretation.

However, as you probably guessed, SI doesn’t believe that you need Big Data to overcome these traps. Big Brains will suffice. As the authors note, a curious open mind who does her research can overcome each of these traps. An open mind who realizes she doesn’t know much about a market and dives into it, reviewing local research and local media, will get a more realistic perspective than one which makes snap decisions based upon his limited experience. An analytical mind that looks at the ratio of success to failures in sales efforts will quickly see that the close rate is way too low, something is wrong, and a thorough investigation is needed. And an inquisitive mind that asks if the market has been thoroughly covered will realize that studies and data that only cover urban centres don’t cover the population as a whole and additional research into non-urban lifestyles, tastes, and buying patterns is needed if the company wishes to reach that market.

Big Data is needed if you want a reasonable chance of accurately predicting the weather beyond the next 24 hours, modelling stresses on a spacecraft under different adverse conditions, or brute-force breaking the SHA-256 algorithm. It’s typically not needed to get a good handle on a potential market and good product design. Brains effectively put to use will do just fine in these situations, as they have for hundreds of years.