What are common statistical sins? I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of "Statistical Sins", tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc.) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of statistical sins across disciplines. Please, submit one "sin" per answer.
 A: Requesting, and perhaps obtaining The Flow Chart: That graphical thing where you say what the level of your variables are and what sort of relationship you're looking for, and you follow the arrows down to get a Brand Name Test or a Brand Name Statistic.  Sometimes offered with mysterious 'parametric' and 'non-parametric' paths.
A: Using pie charts to illustrate relative frequencies. More here.
A: Using statistics/probability in hypothesis testing to measure the "absolute truth".  Statistics simply cannot do this, they can only be of use in deciding between alternatives, which must be specified from "outside" the statistical paradigm.  Statements such as "the null hypothesis is proved true by the statistics" are just incorrect; statistics can only tell you "the null hypothesis is favoured by the data, compared to the alternative hypothesis".  If you then assume that either the null hypothesis or the alternative must be true, you can say "the null proved true", but this is only a trivial consequence of your assumption, not anything demonstrated by the data.
A: Repeating the same or similar experiment over 20 times on the same data and then reporting a statistically significant result with $\alpha = 0.05$.  Incidentally there is a comic about this one.
And similarly to (or almost the same as) @ogrisel's answer, performing a Grid search and reporting only the best result.
A: (With a bit of luck this will be controversial.)
Using a Neyman-Pearson approach to statistical analysis of scientific experiments. Or, worse, using an ill-defined hybrid of Neyman-Pearson and Fisher.
A: Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?
A: The most dangerous trap I encountered when working on a predictive model is not to reserve a test dataset early on so as to dedicate it to the "final" performance evaluation.
It's really easy to overestimate the predictive accuracy of your model if you have a chance to somehow use the testing data when tweaking the parameters, selecting the prior, selecting the learning algorithm stopping criterion...
To avoid this issue, before starting your work on a new dataset you should split your data as:


*

*development set

*evaluation set


Then split your development set as a "training development set" and "testing development set" where you use the training development set to train various models with different parameters and select the bests according to there performance on the testing development set. You can also do grid search with cross validation but only on the development set. Never use the evaluation set while model selection is not 100% done.
Once you are confident with the model selection and parameters, perform a 10 folds cross-validation on the evaluation set to have an idea of the "real" predictive accuracy of the selected model.
Also if your data is temporal, it is best to choose the development / evaluation split on a time code: "It's hard to make predictions - especially about the future."
A: In psychology, the cardinal sin (for me) is the use of principal components analysis to examine the hypothesised latent structure underlying a psychometric test. 
Not testing for normality before using tests which assume this. 
A: Temptation to use advanced statistical methods without understanding them, just because they sound impressive or because they happen to better support researcher's initial hypothesis. 
When one uses an advanced method he or she should have solid reasons as to why the method is appropriate.
A: Reporting p-values when you did data-mining (hypothesis discovery) instead of statistics (hypothesis testing).
A: Probably not as applicable to psych stats (or is it? I'm not sure) but failing to account for a split plot design in an analysis of an experiment.  I've seem way too many people do this.
A: I would say, doing tests and regressions on a small set of data.
Edit: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.
A: Rush into modeling before spending enough time on understanding and preprocessing the data.
A: A few mistakes that bother me:


*

*Assuming unbiased estimators are always better than biased estimators.

*Assuming that a high $R^2$ implies a good model, low $R^2$ implies a bad model.

*Incorrectly interpreting/applying correlation.

*Reporting point estimates without standard error.

*Using methods which assume some sort of Multivariate Normality (such as Linear Discriminant Analysis) when more robust, better performing, non/semiparametric methods are available. 

*Using p-value as a measure of strength between a predictor and the response, rather than as a measure of how much evidence there is of some relationship. 
A: Testing the hypotheses $H_0:  \mu=0$ versus $H_1: \mu\neq 0$
 (for example in a Gaussian setting)
to justify that $\mu=0$ in a model (i.e mix "$H_0$ is not rejected" and "$H_0$ is true").
A very good example of that type of (very bad) reasoning is when you test whether the variances of two Gaussians are equal (or not) before testing if their mean are equal or not with the assumption of equal variance. 
Another example occurs when you test normality (versus non normality) to justify normality. Every statistician has done that in is life ? it is baaad :) (and should push people to check robustness to non Gaussianity) 
A: Not really answering the question, but there's an entire book on this subject:
Phillip I. Good, James William Hardin (2003). Common errors in statistics (and how to avoid them). Wiley. ISBN 9780471460688
A: interpreting Probability(data | hypothesis) as Probability(hypothesis | data) without the application of Bayes' theorem.
A: Ritualized Statistics.
This "sin" is when you apply whatever thing you were taught, regardless of its appropriateness, because it's how things are done. It's statistics by rote, one level above letting the machine choose your statistics for you.
Examples are Intro to Statistics-level students trying to make everything fit into their modest t-test and ANOVA toolkit, or any time one finds oneself going "Oh, I have categorical data, I should use X" without ever stopping to look at the data, or consider the question being asked.
A variation on this sin involves using code you don't understand to produce output you only kind of understand, but know "the fifth column, about 8 rows down" or whatever is the answer you're supposed to be looking for.
A: Dichotomization of a continuous predictor variable to either "simplify" analysis or to solve for the "problem" of non-linearity in the effect of the continuous predictor.
A: Application of least-squares minimization when maximum-likelihood procedures exist.
A: Using Analysis of Covariance (ANCOVA) to try to "control for" or "regress out" the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: Checking factor/covariate independence in ANCOVA
A: Interpreting a $100\alpha \%$ Confidence Interval $I$ as the probability of finding the "real" parameter inside the interval. 
The most common case is when someone calculates this C.I. ($I$) and interprets the number $\alpha$ as the probability of finding the "true mean" say, $\mu$, inside the interval, i.e., interpreting the C.I. as $P(\mu \in I)=\alpha$.
A: Over-interpreting OLS regression in the presence of known outliers.  If you know that there are particular data in your dataset which are generated by a different process to the process that generates most of the data, and this different process generates wildly different results which show up as outliers, then you have to be very careful in interpreting the model output because the outliers often do substantially move the OLS results.  That's not to say OLS is bad, just that you need to think about the data when interpreting the results.
What's worse is that we often have "never throw away outliers" as common advice to early students. Sometimes it translates into an attitude of keeping the data, warts and all, without really discussing anomalies and outliers.  
Better advice might be: "use a mixture model" or "use Huber/quantile-based/other robust techniques" or "go Bayes and use a hierarchical model".  But everyone should at least learn to just "reanalyse without the suspect outliers and print both analyses and show us a plot" or even "talk qualitatively for a bit about outliers in the conclusion of your paper and suggest it might be a good idea to redo the experiment with fewer foul-ups".
A: Interpreting a statistically significant result as "meaningfully large".
A: This isn't generally considered a sin but I hope it will be one day: using a bad model that doesn't describe reality just because it's "interpretable."
A: Maybe stepwise regression and other forms of testing after model selection.
Selecting independent variables for modelling without having any a priori hypothesis behind the existing relationships can lead to logical fallacies or spurious correlations, among other mistakes.
Useful references (from a biological/biostatistical perspective):


*

*Kozak, M., & Azevedo, R. (2011). Does using stepwise variable selection to build sequential path analysis models make sense? Physiologia plantarum, 141(3), 197–200. doi:10.1111/j.1399-3054.2010.01431.x

*Whittingham, M. J., Stephens, P., Bradbury, R. B., & Freckleton, R. P. (2006). Why do we still use stepwise modelling in ecology and behaviour? The Journal of animal ecology, 75(5), 1182–9. doi:10.1111/j.1365-2656.2006.01141.x

*Frank Harrell, Regression Modeling Strategies, Springer 2001.
A: Something I see a surprising amount in conference papers and even journals is making multiple comparisons (e.g. of bivariate correlations) and then reporting all the p<.05s as "significant" (ignoring the rightness or wrongness of that for the moment).
I know what you mean about psychology graduates, as well- I've finished a PhD in psychology and I'm still only just learning really. It's quite bad, I think psychology needs to take quantitative data analysis more seriously if we're going to use it (which, clearly, we should)
A: Not paying attention to levels of measurement, and treating polytomous nominal scales as though they were ordinal, interval, or ratio scales (Ouch).
A: Completely forgetting about checking calibration or normalization, when datasets come for different sensors, different times, different observers.
A: Specifically in psychology, and even more so in marketing, the technique of partial least squares (PLS) is used to "fit" structural equation models and path models, despite being deficient on almost any imaginable performance metric. See McIntosh, Edwards and Antonakis (2014 ORM), Rönkkö, McIntosh and Antonakis (2015 PID) and/or Rönkkö, McIntosh, Antonakis and Edwards (2016 JOM) for detailed treatment, including spelling out some natural requirements like bias and consistency, and demonstration of how PLS fails them (compared to other regression-type methods such as Bollen's model-implied instrumental variables). (I don't know how substitutable the papers are for one another, though; they must cover very similar topics, but may be aimed at somewhat different audiences.)
A: Being exploratory but pretending to be confirmatory. This can happen when one is modifying the analysis strategy (i.e. model fitting, variable selection and so on) data driven or result driven but not stating this openly and then only reporting the "best" (i.e. with smallest p-values) results as if it had been the only analysis. This also pertains to the point if multiple testing that Chris Beeley made and results in a high false positive rate in scientific reports.
A: The one that I see quite often and always grinds my gears is the assumption that a statistically significant main effect in one group and a non-statistically significant main effect in another group implies a significant effect x group interaction.
A: Correlation implies causation, which is not as bad as accepting the Null Hypothesis.
A: Especially in epidemiology and public health - using arithmetic instead of logarithmic scale when reporting graphs of relative measures of association (hazard ratio, odds ratio or risk ratio).
More information here.
A: Analysis of rate data (accuracy, etc) using ANOVA, thereby assuming that rate data has Gaussian distributed error when it's actually binomially distributed.
Dixon (2008) provides a discussion of the consequences of this sin and exploration of more appropriate analysis approaches.
A: Using technical replication instead of true replication, and similarly, using MSE as the denominator in a nested ANOVA F-statistic.
A: A current popular one is plotting 95% confidence intervals around the raw performance values in repeated measures designs when they only relate to the variance of an effect.  For example, a plot of reaction times in a repeated measures design with confidence intervals where the error term is derived from the MSE of a repeated measures ANOVA.  These confidence intervals don't represent anything sensible.  They certainly don't represent anything about the absolute reaction time.  You could use the error term to generate confidence intervals around the effect but that is rarely done.
A: While I can relate to much of what Michael Lew says, abandoning p-values in favor of likelihood ratios still misses a more general problem--that of overemphasizing probabilistic results over effect sizes, which are required to give a result substantive meaning.  This type of error comes in all shapes and sizes and I find it to be the most insidious statistical mistake.  Drawing on J. Cohen and M. Oakes and others, I've written a piece on this at here.
A: My intro psychometrics course in undergrad spent at least two weeks teaching how to perform a stepwise regression. Is there any situation where stepwise regression is a good idea?
A: Failing to test the assumption that error is normally distributed and has constant variance between treatments.  These assumptions aren't always tested, thus least-squares model fitting is probably often used when it is actually inappropriate.
A: This may be more of a pop-stats answer than what you're looking for, but:
Using the mean as an indicator of location when data is highly skewed.
This isn't necessarily a problem, if you and your audience knows what you're talking about, but this generally isn't the case, and the median is often likely to give a better idea of what's going on.
My favourite example is mean wages, which are usually reported as "average wages". Depending on the income/wealth inequality in a country, this can be vastly different from the median wage, which gives a much better indicator for where people are at in real life. For example, in Australia, where we have relatively low inequality, the median is 10-15% lower than the mean. In the US the difference is much starker, the median is less than 70% of the mean, and the gap is increasing.
Reporting on the "average" (mean) wage results in a rosier picture than is warranted, and could also give a large number of people the false impression that they aren't earning as much as "normal" people.
A: Failing to look at (plot) the data.
A: My old stats prof had a "rule of thumb" for dealing with outliers: If you see an outlier on your scatterplot, cover it up with your thumb :)
A: Most interpretations of p-values are sinful! The conventional usage of p-values is badly flawed; a fact that, in my opinion, calls into question the standard approaches to the teaching of hypothesis tests and tests of significance.
Haller and Krause have found that statistical instructors are almost as likely as students to misinterpret p-values. (Take the test in their paper and see how you do.) Steve Goodman makes a good case for discarding the conventional (mis-)use of the p -value in favor of likelihoods. The Hubbard paper is also worth a look.
Haller and Krauss. Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research (2002) vol. 7 (1) pp. 1-20 (PDF)
Hubbard and Bayarri. Confusion over Measures of Evidence (p's) versus Errors (α's) in Classical Statistical Testing. The American Statistician (2003) vol. 57 (3)
Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med (1999) vol. 130 (12) pp. 995-1004 (PDF)
Also see:
Wagenmakers, E-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779-804.
for some clear cut cases where even the nominally "correct" interpretation of a p-value has been made incorrect due to the choices made by the experimenter.
Update (2016): In 2016, American Statistical Association issued a statement on p-values, see here. This was, in a way, a response to the "ban on p-values" issued by a psychology journal about a year earlier.
A: That the p-value is the probability that the null hypothesis is true and (1-p) is the probability that the alternative hypothesis is true, of that failing to reject the null hypothesis means the alternative hypothesis is false etc.
A: In similar vein to @dirkan - The use of p-values as a formal measure of evidence of the null hypothesis being true.  It does have some good heuristic and intuitively good features, but is essentially an incomplete measure of evidence because it makes no reference to the alternative hypothesis.  While the data may be unlikely under the null (leading to a small p-value), the data may be even more unlikely under the alternative hypothesis.
The other problem with p-values, which also relates to some styles of hypothesis testing, is there is no principle telling you which statistic you should choose, apart from the very vague "large value" $\rightarrow$ "unlikely if null hypothesis is true".  Once again, you can see the incompleteness showing up, for you should also have "large value" $\rightarrow$ "likely if alternative hypothesis is true" as an additional heuristic feature of the test statistic.
