# What are common statistical sins?

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of Statistical Sins across disciplines. Please submit one "sin" per answer.

-
I'm aware that "sin" is possibly inflammatory and that that some aspects of statistical analysis are not black-and-white. My intention is to solicit cases where a given commonly-taught practice is pretty clearly inappropriate. – Mike Lawrence Nov 15 '10 at 18:53
You can also add biology/life sciences students to the mix if you like ;) – nico Nov 15 '10 at 19:03
possible duplicate of What are common statistical sins? – whuber Feb 4 '11 at 14:46

Requesting, and perhaps obtaining The Flow Chart: That graphical thing where you say what the level of your variables are and what sort of relationship you're looking for, and you follow the arrows down to get a Brand Name Test or a Brand Name Statistic. Sometimes offered with mysterious 'parametric' and 'non-parametric' paths.

-

Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?

-

Repeating the same or similar experiment over 20 times on the same data and then reporting a statistically significant result with $\alpha = 0.05$. Incidentally there is a comic about this one.

And similarly to (or almost the same as) @ogrisel's answer, performing a Grid search and reporting only the best result.

-

This may be more of a pop-stats answer than what you're looking for, but:

Using the mean as an indicator of location when data is highly skewed.

This isn't necessarily a problem, if you and your audience knows what you're talking about, but this generally isn't the case, and the median is often likely to give a better idea of what's going on.

My favourite example is mean wages, which are usually reported as "average wages". Depending on the income/wealth inequality in a country, this can be vastly different from the median wage, which gives a much better indicator for where people are at in real life. For example, in Australia, where we have relatively low inequality, the median is 10-15% lower than the mean. In the US the difference is much starker, the median is less than 70% of the mean, and the gap is increasing.

Reporting on the "average" (mean) wage results in a rosier picture than is warranted, and could also give a large number of people the false impression that they aren't earning as much as "normal" people.

-
show 1 more comment

I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.

-
Perhaps I don't see why this is such a problem. Hypothesis testing a small sample size using a normal distribution, sure, but using a more conservative/nonparametric test, is this so bad? – Christopher Aden Nov 16 '10 at 10:18

Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA

-

Over-interpreting OLS regression in the presence of known outliers. If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results. That's not to say OLS is bad, just that you need to think about the data when interpreting the results.

What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.

Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model". But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".

-
show 1 more comment

A very good paper on this topic from the econometrics-sphere is Kennedy (2002): Sinning in the Basement: What are the Rules? The Ten Commandments of Applied Econometrics

-

http://sqab.psychology.org/movies/Grace01High.wmv

This video was highly informative for me in the use and interpretation of statistical tests.

-
Can you say something about the topic of this video, so that people don't have to watch it first to know what it's about & whether it was worth their time? – gung Aug 14 '12 at 18:07

Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval.

The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.

-
Is this such a bad thing? I understand the usual argument of "the mean is either in the interval or it isn't", or at least I think I do. But the endpoints of the confidence interval are random variables, so why is it wrong to talk about the probability that they take values above and below the true mean? – mark999 Apr 8 '12 at 22:43
Thanks for the explanation, I think I understand now. I wasn't suggesting that a particular realisation of a 95% confidence interval has 95% chance of containing the true value, although I didn't make that clear. What I meant was that saying "the probability that the (generic) interval contains the true value is 0.95" seems to me to be equivalent to saying "if repeated many times, 95% of the intervals will contain the true value". – mark999 Apr 9 '12 at 4:55