227
$\begingroup$

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc.) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of statistical sins across disciplines. Please, submit one "sin" per answer.

$\endgroup$
  • 5
    $\begingroup$ I'm aware that "sin" is possibly inflammatory and that that some aspects of statistical analysis are not black-and-white. My intention is to solicit cases where a given commonly-taught practice is pretty clearly inappropriate. $\endgroup$ – Mike Lawrence Nov 15 '10 at 18:53
  • 5
    $\begingroup$ You can also add biology/life sciences students to the mix if you like ;) $\endgroup$ – nico Nov 15 '10 at 19:03
  • 1
    $\begingroup$ maybe retitle it life science statistical sins?... or something else more specific... $\endgroup$ – John Nov 15 '10 at 19:27
  • 1
    $\begingroup$ @whuber There was some good answers, so I've merged them both. $\endgroup$ – user88 Feb 6 '11 at 11:02
  • 1
    $\begingroup$ Hi @Amanda, could you give some indication here of what's in the talk? No-one likes the possibility of being rick-rolled. $\endgroup$ – naught101 Oct 21 '12 at 2:17

46 Answers 46

8
$\begingroup$

Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?

$\endgroup$
  • 4
    $\begingroup$ The situation in developed countries is exactly the same. $\endgroup$ – Flounderer Aug 17 '14 at 1:41
7
$\begingroup$

In psychology, the cardinal sin (for me) is the use of principal components analysis to examine the hypothesised latent structure underlying a psychometric test.

Not testing for normality before using tests which assume this.

$\endgroup$
  • 1
    $\begingroup$ Take a look at the sin I have already proposed about testing. If you do the test and do not reject normality, it does not mean that you have a normal sample... it only means that you cannot say the sample is not normal. sin ! $\endgroup$ – robin girard Dec 1 '10 at 16:04
  • $\begingroup$ stats.stackexchange.com/questions/2492/… ... I especially recommend Harvey's answer there. $\endgroup$ – Glen_b -Reinstate Monica Oct 18 '17 at 4:41
7
$\begingroup$

Temptation to use advanced statistical methods without understanding them, just because they sound impressive or because they happen to better support researcher's initial hypothesis.

When one uses an advanced method he or she should have solid reasons as to why the method is appropriate.

$\endgroup$
6
$\begingroup$

Probably not as applicable to psych stats (or is it? I'm not sure) but failing to account for a split plot design in an analysis of an experiment. I've seem way too many people do this.

$\endgroup$
  • $\begingroup$ A pre-post, experimental-control group design is extremely common in psychology. I agree that few people seem to be aware of the comparatively complicated and strict assumptions. Mixed models often seem to be beyond the statistical horizon. $\endgroup$ – caracal Dec 1 '10 at 21:36
5
$\begingroup$

I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.

$\endgroup$
  • 4
    $\begingroup$ Perhaps I don't see why this is such a problem. Hypothesis testing a small sample size using a normal distribution, sure, but using a more conservative/nonparametric test, is this so bad? $\endgroup$ – Christopher Aden Nov 16 '10 at 10:18
  • $\begingroup$ I agree that using a more conservative model to fit the data is the best we can do. But in any case you will have to trust this model. It will be a fitting, not a model. A model requires a representative set of data otherwise it may not work in the future. $\endgroup$ – RockScience Nov 17 '10 at 2:17
  • $\begingroup$ If you use Bayesian regression then the error bars also indicate the uncertainty due to the finite nature of the dataset (given the prior), you only trust the model as far as the error bars suggest you ought to trust it. If you don't have enough data to make a useful inference it will generally be evident in the posterior distributions for the parameters and/or predictions. The usual frequentist error bars will probably say pretty much the same thing. At the end of the day, sometimes only a small dataset is available, it just limits the confidence in your conlcusions. $\endgroup$ – Dikran Marsupial Nov 19 '10 at 9:14
  • $\begingroup$ Agreed for the Bayesian regression. Thanks for pointing that out. But if you have two points that form a straight line, how do you calculate the frequentist error bars? And let's say that you have enough points to calculate the frequentist error bars, from how many points can you trust them (should we use the error bars of the error bars?) $\endgroup$ – RockScience Nov 19 '10 at 9:26
  • 1
    $\begingroup$ Isn't performing a test implicitly looking at the confidence intervals? Perhaps it would be that the sin test-wise is ignoring the power of the test? $\endgroup$ – Dikran Marsupial Nov 19 '10 at 13:11
5
$\begingroup$

Rush into modeling before spending enough time on understanding and preprocessing the data.

$\endgroup$
  • $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review $\endgroup$ – kjetil b halvorsen Nov 28 '16 at 8:26
  • $\begingroup$ This is probably adequate, IMO, given the context of the thread to which it was posted. I think it's OK. @Aliweb, you might want to develop / elaborate the idea a little, though. $\endgroup$ – gung - Reinstate Monica Nov 28 '16 at 13:06
  • $\begingroup$ @kjetilbhalvorsen This exactly answers the question. The author is asking for some statistical sins and if you read the question, my answer couldn't be seen as a critic to the author. $\endgroup$ – Aliweb Nov 28 '16 at 17:42
  • 1
    $\begingroup$ @gung I tried to answer the question the same way many others have done and considering what the asker is asking for. I think elaborating the idea would be kind of repeating myself or I should start talking about "how to understand/preprocess the data" which is not what the question is about. $\endgroup$ – Aliweb Nov 28 '16 at 17:44
  • 1
    $\begingroup$ @Aliweb, that's fair. I only said "might"; if you can't go further w/o repeating yourself or moving off topic, then you're best staying put. $\endgroup$ – gung - Reinstate Monica Nov 28 '16 at 18:31
4
$\begingroup$

Application of least-squares minimization when maximum-likelihood procedures exist.

$\endgroup$
  • 3
    $\begingroup$ Would you please explain the consequences of this sin? $\endgroup$ – russellpierce Nov 16 '10 at 2:50
  • 1
    $\begingroup$ If the data are generated by a process with a heteroscedastic noise process, the regression model is likely to give very inacurate out-of-sample predictions. $\endgroup$ – Dikran Marsupial Nov 16 '10 at 9:26
  • $\begingroup$ This entry was originally inspired by the observation that some folks estimate non-linear psychometric functions by minimizing least squares. For example, Murd et al (2009, perceptionweb.com/abstract.cgi?id=p6145 , free pdf available by googling the title) fit a probit function through data by minimizing least-squares. $\endgroup$ – Mike Lawrence Nov 16 '10 at 16:04
  • $\begingroup$ So, should the "answer" be amended to "application of least-squares minimization on heteroscedatic data"? $\endgroup$ – russellpierce Nov 17 '10 at 15:13
  • 3
    $\begingroup$ @MikeLawrence - "maximum likelihood" = "weighted least squares" in many cases. even maximum entropy is approximately least squares (when the initial measure isn't too far from the optimum measure). You could do a lot worse than least squares... $\endgroup$ – probabilityislogic Mar 9 '12 at 5:43
4
$\begingroup$

Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA

$\endgroup$
4
$\begingroup$

Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval.

The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.

$\endgroup$
  • 3
    $\begingroup$ Is this such a bad thing? I understand the usual argument of "the mean is either in the interval or it isn't", or at least I think I do. But the endpoints of the confidence interval are random variables, so why is it wrong to talk about the probability that they take values above and below the true mean? $\endgroup$ – mark999 Apr 8 '12 at 22:43
  • $\begingroup$ It IS bad because you are giving a probability statement that doesn't exist. The phrase "the true population mean $\mu$ is either in the interval or isn't" states a beautiful fact: $P(\mu \in I)$ is either 1 or 0. On the other hand, recall that what you calculate when creating confidence intervals is $P(L_1<\bar{X}-\mu<L_2)=\alpha$, where $\bar{X}$ is a STATISTIC (i.e. a function of random variables) and is, therefore, the random variable on which you are calculating the probability. The 100$\alpha$% confidence interval that you calculate is one of many other (random) intervals that... $\endgroup$ – Néstor Apr 9 '12 at 3:58
  • $\begingroup$ ...might "appear" as you sample from the population. Simply put, $I$ is one realization of many random intervals (say, $I_r$) that are generated as you sample from the population. You can only say that $I_r$ is a range in which the mean will occur 95% of the time, but that says nothing about your particular realization (or estimation) of interval $I$ that one usually calculates, and, therefore, says nothing about the probability of $I$ containing $\mu$. $\endgroup$ – Néstor Apr 9 '12 at 4:06
  • 2
    $\begingroup$ Thanks for the explanation, I think I understand now. I wasn't suggesting that a particular realisation of a 95% confidence interval has 95% chance of containing the true value, although I didn't make that clear. What I meant was that saying "the probability that the (generic) interval contains the true value is 0.95" seems to me to be equivalent to saying "if repeated many times, 95% of the intervals will contain the true value". $\endgroup$ – mark999 Apr 9 '12 at 4:55
  • $\begingroup$ Oh, yes! That is in fact true :-). Just a misunderstanding then. $\endgroup$ – Néstor Apr 9 '12 at 4:57
4
$\begingroup$

Over-interpreting OLS regression in the presence of known outliers. If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results. That's not to say OLS is bad, just that you need to think about the data when interpreting the results.

What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.

Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model". But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".

$\endgroup$
  • $\begingroup$ Hah. I already added this as a comment to my answer, but it's more relevant to this answer: tamino.wordpress.com/2012/03/29/… discusses situations where OLS may be better than robust regression, even in cases with (apparent) outliers. $\endgroup$ – naught101 Apr 11 '12 at 1:28
4
$\begingroup$

Interpreting a statistically significant result as "meaningfully large".

$\endgroup$
4
$\begingroup$

This isn't generally considered a sin but I hope it will be one day: using a bad model that doesn't describe reality just because it's "interpretable."

$\endgroup$
3
$\begingroup$

Completely forgetting about checking calibration or normalization, when datasets come for different sensors, different times, different observers.

$\endgroup$
3
$\begingroup$

Specifically in psychology, and even more so in marketing, the technique of partial least squares (PLS) is used to "fit" structural equation models and path models, despite being deficient on almost any imaginable performance metric. See McIntosh, Edwards and Antonakis (2014 ORM), Rönkkö, McIntosh and Antonakis (2015 PID) and/or Rönkkö, McIntosh, Antonakis and Edwards (2016 JOM) for detailed treatment, including spelling out some natural requirements like bias and consistency, and demonstration of how PLS fails them (compared to other regression-type methods such as Bollen's model-implied instrumental variables). (I don't know how substitutable the papers are for one another, though; they must cover very similar topics, but may be aimed at somewhat different audiences.)

$\endgroup$
  • $\begingroup$ Didn't you just list a different paper from the chemometrics literature somewhere? It might be worth adding to the list. $\endgroup$ – gung - Reinstate Monica Aug 11 '16 at 20:00
  • $\begingroup$ Good point -- it was @amoeba who was familiar with the chemometrics literature; I am not qualified to speak on that. My understanding is that it is used with much less fanfare there than in psych and marketing, just to honestly reduce dimensionality rather than to claim that you found some underlying factors and quantified them perfectly. $\endgroup$ – StasK Aug 11 '16 at 21:08
3
$\begingroup$

Not paying attention to levels of measurement, and treating polytomous nominal scales as though they were ordinal, interval, or ratio scales (Ouch).

$\endgroup$
2
$\begingroup$

Using technical replication instead of true replication, and similarly, using MSE as the denominator in a nested ANOVA F-statistic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.