What are common statistical sins?

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc.) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of statistical sins across disciplines. Please, submit one "sin" per answer.

• I'm aware that "sin" is possibly inflammatory and that that some aspects of statistical analysis are not black-and-white. My intention is to solicit cases where a given commonly-taught practice is pretty clearly inappropriate. Nov 15, 2010 at 18:53
• You can also add biology/life sciences students to the mix if you like ;)
– nico
Nov 15, 2010 at 19:03
• maybe retitle it life science statistical sins?... or something else more specific...
– John
Nov 15, 2010 at 19:27
• @whuber There was some good answers, so I've merged them both.
– user88
Feb 6, 2011 at 11:02
• Hi @Amanda, could you give some indication here of what's in the talk? No-one likes the possibility of being rick-rolled. Oct 21, 2012 at 2:17

Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?

• The situation in developed countries is exactly the same. Aug 17, 2014 at 1:41

In psychology, the cardinal sin (for me) is the use of principal components analysis to examine the hypothesised latent structure underlying a psychometric test.

Not testing for normality before using tests which assume this.

• Take a look at the sin I have already proposed about testing. If you do the test and do not reject normality, it does not mean that you have a normal sample... it only means that you cannot say the sample is not normal. sin ! Dec 1, 2010 at 16:04
• stats.stackexchange.com/questions/2492/… ... I especially recommend Harvey's answer there. Oct 18, 2017 at 4:41

Temptation to use advanced statistical methods without understanding them, just because they sound impressive or because they happen to better support researcher's initial hypothesis.

When one uses an advanced method he or she should have solid reasons as to why the method is appropriate.

Probably not as applicable to psych stats (or is it? I'm not sure) but failing to account for a split plot design in an analysis of an experiment. I've seem way too many people do this.

• A pre-post, experimental-control group design is extremely common in psychology. I agree that few people seem to be aware of the comparatively complicated and strict assumptions. Mixed models often seem to be beyond the statistical horizon. Dec 1, 2010 at 21:36

I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.

• Perhaps I don't see why this is such a problem. Hypothesis testing a small sample size using a normal distribution, sure, but using a more conservative/nonparametric test, is this so bad? Nov 16, 2010 at 10:18
• I agree that using a more conservative model to fit the data is the best we can do. But in any case you will have to trust this model. It will be a fitting, not a model. A model requires a representative set of data otherwise it may not work in the future. Nov 17, 2010 at 2:17
• If you use Bayesian regression then the error bars also indicate the uncertainty due to the finite nature of the dataset (given the prior), you only trust the model as far as the error bars suggest you ought to trust it. If you don't have enough data to make a useful inference it will generally be evident in the posterior distributions for the parameters and/or predictions. The usual frequentist error bars will probably say pretty much the same thing. At the end of the day, sometimes only a small dataset is available, it just limits the confidence in your conlcusions. Nov 19, 2010 at 9:14
• Agreed for the Bayesian regression. Thanks for pointing that out. But if you have two points that form a straight line, how do you calculate the frequentist error bars? And let's say that you have enough points to calculate the frequentist error bars, from how many points can you trust them (should we use the error bars of the error bars?) Nov 19, 2010 at 9:26
• Isn't performing a test implicitly looking at the confidence intervals? Perhaps it would be that the sin test-wise is ignoring the power of the test? Nov 19, 2010 at 13:11

Rush into modeling before spending enough time on understanding and preprocessing the data.

• This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review Nov 28, 2016 at 8:26
• This is probably adequate, IMO, given the context of the thread to which it was posted. I think it's OK. @Aliweb, you might want to develop / elaborate the idea a little, though. Nov 28, 2016 at 13:06
• @kjetilbhalvorsen This exactly answers the question. The author is asking for some statistical sins and if you read the question, my answer couldn't be seen as a critic to the author. Nov 28, 2016 at 17:42
• @gung I tried to answer the question the same way many others have done and considering what the asker is asking for. I think elaborating the idea would be kind of repeating myself or I should start talking about "how to understand/preprocess the data" which is not what the question is about. Nov 28, 2016 at 17:44
• @Aliweb, that's fair. I only said "might"; if you can't go further w/o repeating yourself or moving off topic, then you're best staying put. Nov 28, 2016 at 18:31

Application of least-squares minimization when maximum-likelihood procedures exist.

• Would you please explain the consequences of this sin? Nov 16, 2010 at 2:50
• If the data are generated by a process with a heteroscedastic noise process, the regression model is likely to give very inacurate out-of-sample predictions. Nov 16, 2010 at 9:26
• This entry was originally inspired by the observation that some folks estimate non-linear psychometric functions by minimizing least squares. For example, Murd et al (2009, perceptionweb.com/abstract.cgi?id=p6145 , free pdf available by googling the title) fit a probit function through data by minimizing least-squares. Nov 16, 2010 at 16:04
• So, should the "answer" be amended to "application of least-squares minimization on heteroscedatic data"? Nov 17, 2010 at 15:13
• @MikeLawrence - "maximum likelihood" = "weighted least squares" in many cases. even maximum entropy is approximately least squares (when the initial measure isn't too far from the optimum measure). You could do a lot worse than least squares... Mar 9, 2012 at 5:43

Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA

Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval.

The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.

• Is this such a bad thing? I understand the usual argument of "the mean is either in the interval or it isn't", or at least I think I do. But the endpoints of the confidence interval are random variables, so why is it wrong to talk about the probability that they take values above and below the true mean? Apr 8, 2012 at 22:43
• It IS bad because you are giving a probability statement that doesn't exist. The phrase "the true population mean $\mu$ is either in the interval or isn't" states a beautiful fact: $P(\mu \in I)$ is either 1 or 0. On the other hand, recall that what you calculate when creating confidence intervals is $P(L_1<\bar{X}-\mu<L_2)=\alpha$, where $\bar{X}$ is a STATISTIC (i.e. a function of random variables) and is, therefore, the random variable on which you are calculating the probability. The 100$\alpha$% confidence interval that you calculate is one of many other (random) intervals that... Apr 9, 2012 at 3:58
• ...might "appear" as you sample from the population. Simply put, $I$ is one realization of many random intervals (say, $I_r$) that are generated as you sample from the population. You can only say that $I_r$ is a range in which the mean will occur 95% of the time, but that says nothing about your particular realization (or estimation) of interval $I$ that one usually calculates, and, therefore, says nothing about the probability of $I$ containing $\mu$. Apr 9, 2012 at 4:06
• Thanks for the explanation, I think I understand now. I wasn't suggesting that a particular realisation of a 95% confidence interval has 95% chance of containing the true value, although I didn't make that clear. What I meant was that saying "the probability that the (generic) interval contains the true value is 0.95" seems to me to be equivalent to saying "if repeated many times, 95% of the intervals will contain the true value". Apr 9, 2012 at 4:55
• Oh, yes! That is in fact true :-). Just a misunderstanding then. Apr 9, 2012 at 4:57

Over-interpreting OLS regression in the presence of known outliers. If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results. That's not to say OLS is bad, just that you need to think about the data when interpreting the results.

What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.

Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model". But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".

• Hah. I already added this as a comment to my answer, but it's more relevant to this answer: tamino.wordpress.com/2012/03/29/… discusses situations where OLS may be better than robust regression, even in cases with (apparent) outliers. Apr 11, 2012 at 1:28

Interpreting a statistically significant result as "meaningfully large".

This isn't generally considered a sin but I hope it will be one day: using a bad model that doesn't describe reality just because it's "interpretable."

Completely forgetting about checking calibration or normalization, when datasets come for different sensors, different times, different observers.

Specifically in psychology, and even more so in marketing, the technique of partial least squares (PLS) is used to "fit" structural equation models and path models, despite being deficient on almost any imaginable performance metric. See McIntosh, Edwards and Antonakis (2014 ORM), Rönkkö, McIntosh and Antonakis (2015 PID) and/or Rönkkö, McIntosh, Antonakis and Edwards (2016 JOM) for detailed treatment, including spelling out some natural requirements like bias and consistency, and demonstration of how PLS fails them (compared to other regression-type methods such as Bollen's model-implied instrumental variables). (I don't know how substitutable the papers are for one another, though; they must cover very similar topics, but may be aimed at somewhat different audiences.)

• Didn't you just list a different paper from the chemometrics literature somewhere? It might be worth adding to the list. Aug 11, 2016 at 20:00
• Good point -- it was @amoeba who was familiar with the chemometrics literature; I am not qualified to speak on that. My understanding is that it is used with much less fanfare there than in psych and marketing, just to honestly reduce dimensionality rather than to claim that you found some underlying factors and quantified them perfectly. Aug 11, 2016 at 21:08

Not paying attention to levels of measurement, and treating polytomous nominal scales as though they were ordinal, interval, or ratio scales (Ouch).

Using technical replication instead of true replication, and similarly, using MSE as the denominator in a nested ANOVA F-statistic.