Big Data = Big Mistake

FT.com recently published a great article on Big Data that asked Are We Making a Big Mistake which contains the best description SI has seen yet for Big Data: Big Mistake!

Why? Because, even though there are times we might want correlation to be causation (because then we could put an end to IE once and for all), it is not, never was, and never will be. never, Ever, EVER!* And, as pointed out in the article, just because a correlation algorithm works great for predicting trends, such as the spread of influenza, three years in a row, this doesn’t mean it’s going to work well the fourth year. Randomly identified statistical patterns in data are just that — randomly identified statistical patterns in data.

The Google example in the article is a huge example of how big data can fail in a massive, embarrassing way. In Nature 457, published 19 February 2009, Google published a paper entitled detecting influenza epidemics using search engine query data that detailed how they were able to track the spread of influenza across the US more quickly than the Centers for Disease Control and Prevention (CDC). Using a big data algorithm that detected a correlation between what people searched for and and whether they had flu systems, Google was apparently able to track the spread of influenza with only a day’s delay, compared to the week or more it took the CDC to assemble a picture based on reports from doctors’. This theory free approach worked for four years, and then failed spectacularly in 2013 when it drastically over-estimated peak flu levels, as chronicled in this article on When Google Got Flu Wrong over on Nature.com.

To put the issue of correlation vs causation into terms everyone can understand, if correlation was causation, Microsoft would be on trial as an accomplice to felony murder in every state in the United States, since the declining usage of internet explorer directly correlates with the declining murder rate in the US:

In other words, if correlation was causation, then using Internet Explorer invokes violent tendencies which leads to murder, and its continued existence is criminal.**

This is the problem with big data today. Everyone is using it to try and detect potentially useful correlations, instead of trying to support or disprove useful, actionable, theories. Why? Because, as the FT.com article states, figuring out what causes what is hard, and some would even claim it to be impossible.

Correlation might work in the short term, as it did for Google that was able to predict the spread of influenza for a few years, but it always fails in the long term. And if you have no idea what is behind a correlation, you have no idea what might cause that correlation to break down. Just like a stock market trading algorithm, it might work for a year, a month, a week, a day, or a minute. You just don’t know.

That’s why relying on correlation-based big-data algorithms is a big mistake. While they will give you interesting patterns to examine, relying on them will lead you down a dark and winding road that leads to the edge of a deep canyon (that you are aren’t going to see until you fall in). Unless you can come up with a reasonable theory and support it with the data, it’s just an interesting pattern — and you should continue on your merry way until you find an interesting pattern you can actually explain unless you too want to end up with egg on your face.

That’s why Sourcing Innovation Still Prefers Big Brains to Big Data, and likely always will. We might be slaves to the corporations in the continuum, but that doesn’t mean we have to be slaves to stupidity.

* Everyone should know by now that correlation is not causation given that Pinky and the Brain gave you all a great Lesson in Statistics six years ago (when they were still in the employ of a certain Burlington sourcing provider …)

** It’s distribution was criminal for a while when Microsoft tried to create a browser monopoly by embedding it in the Operating System in a way that led Windows users to believe there was no other choice, as monopolies are illegal in many countries, but, I’m sorry to say, the continued existence of IE is not criminal, just sad and frustrating.

Magic & Logic (i.e. Marketing & Procurement) Through the Years [2014] (Collected Links)

PREAMBLE

Magic & Logic

The Creative Challenge

Efficient Sourcing in Marketing

How Do You Support Marketing and Get a Grip on Agency Lifecycle Management?

Too Many Marketing Fingers in the Procurement Pie?

BONUS

890 Years Ago Today

David I becomes King of Scots. And while one could write a nice treatise about how he was a pious king, a reformer and civilizing agent in a barbarian nation and on the effects of his changes on Scottish cultural development, I’m not going to. (Wikipedia) There are full bodies of work out there you can read if you are interested.

The only reason SI is mentioning his royal coronation 890 years later is because he was the first to bring Scotland into the modern era. Despite the fact that it is England’s northern neighbour, the Scot’s didn’t have their own coinage until David’s reign, even though the Celts (which immigrated from Northern France) were producing coins as early as 80 BC and the Anglo-Saxons, after the departure of the Romans around 450 AD, began to produce their own coinage around 600 AD with the Viking colonization of the north-east of England (that minted coins in York). Charlemagne produced denier in the Kingdom of the Franks around 755 AD, and when Eadgar became king of England in 959, he introduced a silver penny that was used throughout the whole of England. But Scotland didn’t get their own coin for another 170 years!

The modern economy runs on the global banking system, which runs on banknotes, which was a replacement for the copper, silver, and gold coins that defined the monetary system of the time. This was an important first step in Scotland’s economic development, and a reminder of just how young our modern global economy really is when some places, known to be inhabited for millennia, didn’t even have coins a thousand years ago!

But the Scots learned fast. Now they are one of the most fiscally sound countries in Europe and even considering Independence in the fall (in a national referendum on September 18) as they believe they can do better economically on their own than as part of the UK!

Anyone have any thoughts on this?

The (Board) Gamer’s Guide to Supply Management Part XVII: The Village

Life is tough in Competitive Co. Global growth is slowing with the economy. Your products are no longer the most wanted in the marketplace. And at the end of every quarter, the employees who fall in the bottom 10% of their performance reviews get axed. Plus, you just found out that the CPO has been lured away to Big Money Co., one of the Directors is getting promoted, and it’s likely that the next Director will be promoted from within. You want that job, because it significantly decreases the chance that you will be cut when your next annual performance review comes up.

But how do you progress up the corporate ladder? Where do you focus your efforts? Even though you’re required to be a jack of all trades as a Category Manager, you can’t focus on everything, and if you’ll try, you’ll be a master of none and one of your category management rivals will be picked instead of you. Looking around, you see that people who get promoted to Senior Management positions tend to be those who either excel in timing the market and locking in contracts and/or spot buys at the perfect time, forging new markets, managing supplier relationships to optimize production, embedding themselves in standards organizations, or mastering the politics of the workplace to advance despite their lack of skills.

It’s not much different than trying to progress up the social ladder in the typical medieval village. If you were unhappy with the simple life of a farmer (like in Agricola) or a fisherman (like in Le Havre or Rouen Upon a Salty Ocean), then you either have to make your life as a merchant in the market and learn how to profit on every trade, travel to strange new lands to find new and valuable goods for trade, become a master craftsman and produce the ploughs and carriages needed by the farmers and traders, take the religious path and join the church and try to become a monk and work your way up the pecking order, or, if you were lucky to be living in one of the early commonwealths or states, where leaders were elected, become a politician and try to work your way up the ranks in the city council to eventually become elected (or appointed) as the representative of your city.

In other words, things haven’t changed much in 300 years. If you want to get ahead in the world, you either master trade, travel, craft, religion*, or politics. And that’s what you have to do in The Village.
In The Village, you win by controlling the family that achieves the most prestige by the end of the game. Prestige is gained by traveling to new cities, occupying the council chamber(s), progressing through the ranks in the church, succeeding in trade at the market, amassing wealth (in the form of gold coins), and getting your family members recorded in the village chronicle (for their deeds) on their demise. (Unlike the other worker-based games we have covered so far in this Board Gamer’s Guide to Supply Management, where you either have a constant number of workers, or a slowly increasing number of workers as the game progresses, in this game, you lose workers as the game progresses as actions have a time element, and you only get so much time per worker. This adds a new dimension of complexity as you have to be balance worker acquisition with worker loss, just like in the real world where workers retire, defect, or, occasionally, drop dead at their desks.)

As with most worker placement games, the game is round-based and each round consists of a number of actions. In each round, there are a fixed number of birthing, grain harvest, craft, market, travel, council, church, and well actions and players take turns until all the actions have been taken or the end-game has been triggered (in which case each player takes one more action). The birthing action adds one family member (worker) to your family; the grain harvest allows you to take 2 (3, or 4) bags of grain (if you have a plow and a orse or an ox); the craft action allows you to either produce or trade for a scroll, plow, or wagon, mill grain and sell it for gold, or work or trade for a horse or ox; the market action allows you to trade goods (and acquire prestige for your trading skills); the travel action allows you to visit a new city if you have a wagon and the resources to do so; the council action allows you to place a member in the council if you have a scroll or the influence (resources) to do so or, if you have a member in the council, advance him through the ranks if you have the scroll and/or influence (resources) to do so and gain rewards for doing so; the church allows you to enroll one of your family members in the clergy in the hopes that they will be selected for advancement at the end of the round (which will give you prestige if you have the most family members in the clergy); and the well action allows you to acquire a resource (needed for travelling, council advancement, goods, etc.).

The difficulty is in balancing your actions so that you always have the resources and time to advance. For instance, you can’t take too many time-based actions until you have expanded your family as you lose one family member after you have taken actions that require 10 time units in total, and, because this game takes place during the time of the plague, you cannot add more than 7 family members over the duration of the game. Many of the actions, such as travel, advancing through the council chamber, trading in the market, milling grain for gold, and working for scrolls, ploughs, wagons, horses, and oxen, require time. Depending on the action, it can require anywhere from one time unit (to enter the council chambers for the first time) to six time units to produce your first plough (as it takes 3 time units to learn the trade and 3 time units to build a plough). Just like in the real world, it takes time for your workers to learn their jobs and then time to produce results. That’s why you often buy or outsource, trading money and material (in The Village, resources) for goods and services you need instead of trying to acquire the talent and build the product yourself. Some competencies (like category management) you invest in and others (like production) you outsource. Sometimes you keep a superstar on your team, and advance them through the ranks (like you advance them through the council or the church in The Village) and sometimes you let them go (or, in The Village, let them expire to be recorded in the Chronicle for prestige).

It’s another great game for testing your Supply Management muster, with the unique twist that you not only have to balance resources with growth, but you also have to balance trade with workforce output, because, just like in real life, if you burn out your workforce, they expire. Do you have what it takes to be master of The Village. Why don’t you find out? Maybe you’ll even figure out where to focus your efforts to advance your own Supply Management Career!

* Some standards have as many zealots as recognized religions!