I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc.) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of statistical sins across disciplines. Please, submit one "sin" per answer.
Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?
In psychology, the cardinal sin (for me) is the use of principal components analysis to examine the hypothesised latent structure underlying a psychometric test.
Not testing for normality before using tests which assume this.
Temptation to use advanced statistical methods without understanding them, just because they sound impressive or because they happen to better support researcher's initial hypothesis.
When one uses an advanced method he or she should have solid reasons as to why the method is appropriate.
Probably not as applicable to psych stats (or is it? I'm not sure) but failing to account for a split plot design in an analysis of an experiment. I've seem way too many people do this.
I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.
Rush into modeling before spending enough time on understanding and preprocessing the data.
Application of least-squares minimization when maximum-likelihood procedures exist.
Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA
Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval.
The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.
Over-interpreting OLS regression in the presence of known outliers. If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results. That's not to say OLS is bad, just that you need to think about the data when interpreting the results.
What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.
Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model". But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".
Interpreting a statistically significant result as "meaningfully large".
This isn't generally considered a sin but I hope it will be one day: using a bad model that doesn't describe reality just because it's "interpretable."
Completely forgetting about checking calibration or normalization, when datasets come for different sensors, different times, different observers.
Specifically in psychology, and even more so in marketing, the technique of partial least squares (PLS) is used to "fit" structural equation models and path models, despite being deficient on almost any imaginable performance metric. See McIntosh, Edwards and Antonakis (2014 ORM), Rönkkö, McIntosh and Antonakis (2015 PID) and/or Rönkkö, McIntosh, Antonakis and Edwards (2016 JOM) for detailed treatment, including spelling out some natural requirements like bias and consistency, and demonstration of how PLS fails them (compared to other regression-type methods such as Bollen's model-implied instrumental variables). (I don't know how substitutable the papers are for one another, though; they must cover very similar topics, but may be aimed at somewhat different audiences.)
Not paying attention to levels of measurement, and treating polytomous nominal scales as though they were ordinal, interval, or ratio scales (Ouch).
Using technical replication instead of true replication, and similarly, using MSE as the denominator in a nested ANOVA F-statistic.