Lies, Damn Lies, and Statistics

Hopefully you caught The Brain’s much needed lesson in statistics back in January, as it was very informative. (If you didn’t, you can still go back and read it. Heck, even if you did, it probably wouldn’t be a bad idea to go back and read it again.)

The reason I’m pointing it out again is that I just noticed that Knowledge and Wharton put out a great summary of some of the key points in their article on The Use — and Misuse — of Statistics: How and Why Numbers Are So Easily Manipulated. Even though they had to go and use, what is in my view, the waste-of-time, waste-of-print, and waste-of-breath story on Roger Clemens and his alleged (ab)use of steroids as a background, they still made some great points regarding statistics, which need to be reiterated every now and again (because it seems that the vast majority of people who like to do statistical studies and quote statistical results still don’t understand what statistics is really all about).

  • Correlation is not Causation!
    As the article notes, a chain of retail stores may analyze its operations for a set period and find that those times when it reduced its sales prices coincided with times that overall sales fell. The chain might conclude that low prices reduce sales volume when, in fact, it could be the case that stores run semi-annual sales during known down periods. In other words, low sales are causing price declines and not the other way around.
  • It’s much easier to isolate and exclude extraneous data when you have experimental or hard-sciences data.
    In post-activity analysis in a business setting, it’s much more difficult to isolate the effects of a variety of other influences — and any attempt to simplify will most likely lead to incorrect results.
  • Comparing your situation only to those that produced positive effects is selection bias — and it’s wrong! Samples must be random.
    The example the authors use is that the Clemens report tried to prove he didn’t do steroids by noting that there are other examples of professional baseball players, like Nolan Ryan, Randy Johnson, and Curt Schilling, that also enjoyed great success in their 40s. However, that’s atypical behavior. The vast majority of players, and pitchers in particular, steadily improve in their early careers, peak at about 30, and then slowly decline. Clemens started declining in his late 20’s and then rebounded and improved in his 40s.
  • A single, short-term study on a small population is not conclusive! Especially if the population is not representative of the population at large!
    The example given here is a lawsuit filed against the Coca-Cola Company’s marketing for Enviga, it’s caffeinated green-tea drink, that states it actually burns more calories than it provides, resulting in ‘negative calories’. The claim is based on a clinical study of a small group of individuals with an average BMI (Body Mass Index) of 22. However, the majority of the American population has a BMI of 25 or more. Thus, its not statistically reasonable to say that the study would be representative of the population at large.
  • An accounting of the entire testing process is required for proper perspective in interpretation.
    So you found a statistically significant effect, correlation, or difference between some set of variables. If you don’t report the twenty-one insignificant tests you did before you found that one significant result, how do you know it wasn’t a fluke and that the test should probably be repeated?
  • Data-driven studies can’t always tell you the right answer.
    All they can tell you is which answers to eliminate because the data does not support them. The true value of a statistical analysis is that it helps users to properly characterize uncertainty as opposed to a “best guess”, to realize what outcomes are statistically significant, and to answer specific hypotheses.
  • You have to understand what the drivers behind the variables are if you are to have any hope of making a correct interpretation!
    Consider the example of major league baseball outfielders. A hypothesis going into such a study might be that outfielders have a harder time catching balls hit behind them, which forces them to run backwards. You’ll likely find that the opposite was true – that outfielders tend to catch more balls running backwards, even though this seems counter-intuitive at first. However, when you consider the hang-time of the ball, and the fact that balls hit farther are in the air longer, which gives the outfielder more time to catch them, it starts to make sense.
  • The validity of a statistical analysis is only as good as it’s individual components.
    And if even one component is invalid, the whole work is invalid.