What are common statistical sins?

Question

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of Statistical Sins across disciplines. Please submit one "sin" per answer.

Answer 1

Requesting, and perhaps obtaining The Flow Chart: That graphical thing where you say what the level of your variables are and what sort of relationship you're looking for, and you follow the arrows down to get a Brand Name Test or a Brand Name Statistic. Sometimes offered with mysterious 'parametric' and 'non-parametric' paths.

Answer 2

Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?

Answer 3

Repeating the same or similar experiment over 20 times on the same data and then reporting a statistically significant result with $\alpha = 0.05$. Incidentally there is a comic about this one.

And similarly to (or almost the same as) @ogrisel's answer, performing a Grid search and reporting only the best result.

Answer 4

This may be more of a pop-stats answer than what you're looking for, but:

Using the mean as an indicator of location when data is highly skewed.

This isn't necessarily a problem, if you and your audience knows what you're talking about, but this generally isn't the case, and the median is often likely to give a better idea of what's going on.

My favourite example is mean wages, which are usually reported as "average wages". Depending on the income/wealth inequality in a country, this can be vastly different from the median wage, which gives a much better indicator for where people are at in real life. For example, in Australia, where we have relatively low inequality, the median is 10-15% lower than the mean. In the US the difference is much starker, the median is less than 70% of the mean, and the gap is increasing.

Reporting on the "average" (mean) wage results in a rosier picture than is warranted, and could also give a large number of people the false impression that they aren't earning as much as "normal" people.

Answer 5

I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.

Answer 6

Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA

Answer 7

Over-interpreting OLS regression in the presence of known outliers. If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results. That's not to say OLS is bad, just that you need to think about the data when interpreting the results.

What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.

Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model". But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".

Answer 8

A very good paper on this topic from the econometrics-sphere is Kennedy (2002): Sinning in the Basement: What are the Rules? The Ten Commandments of Applied Econometrics

Answer 9

http://sqab.psychology.org/movies/Grace01High.wmv

This video was highly informative for me in the use and interpretation of statistical tests.

Answer 10

Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval.

The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.

Answer 11

I just gave a talk on this subject... A link to the video follows if you are interested.

http://www.youtube.com/watch?v=1SNQQvY1ESo&feature=g-upl

Answer 12

Not paying attention to levels of measurement, and treating polychotomous nominal scales as though they were ordinal, interval, or ratio scales (Ouch).