How to annoy a statistical referee? I recently asked a question regarding general principles around reviewing statistics in papers. What I would now like to ask, is what particularly irritates you when reviewing a paper, i.e. what's the best way to really annoy a statistical referee!
One example per answer, please.
 A: Reporting effects that "approached significance ( p < .10 for example) and then writing about them as though they had attained significance at a more stringent and acceptable level.
Running multiple Structural Equation Models that were not nested and then writing about them as though they were nested.
Taking a well-established analytic strategy and presenting it as though no-one had ever thought of using it before.  Perhaps this qualifies as plagiarism to the nth degree. 
A: What particularly irritates me personally is people who clearly used user-written packages for statistical software but don't cite them properly, or at all, thereby failing to give any credit to the authors. Doing so is particularly important when the authors are in academia and their jobs depend on publishing papers that get cited. (Perhaps I should add that, in my field, many of the culprits are not statisticians.)
A: Goodness me, so many things come to mind...


*

*Stepwise regression

*Splitting continuous data into
groups

*Giving p-values but no measure of
effect size

*Describing data using the mean and
the standard deviation without
indicating whether the data were
more-or-less symmetric and unimodal

*Figures without clear captions (are
those error bars standard errors of
the mean, or standard deviations
within groups, or what?)
A: I recommend the following two articles:
Martin Bland:
How to Upset the Statistical Referee
This is based on a series of talks given by Martin Bland, along with data from other statistical referees (‘a convenience sample with a low response rate’). It ends with an 11-point list of ‘[h]ow to avoid upsetting the statistical referee’.
Stian Lydersen:
Statistical review: frequently given comments
This recent paper (published 2014/2015) lists the author’s 14 most common review comments, based on approx. 200 statistical reviews of scientific papers (in a particular journal). Each comment has a brief explanation of the problem and instructions on how to properly do the analysis/reporting. The list of cited references is a treasure trove of interesting papers.
A: I'm most (and most frequently) annoyed by "validation" aiming at generalization error of predictive models where the test data is not independent (e.g. typically multiple measurements per patient in the data, out-of-bootstrap or cross validation splitting measurements not patients).
Even more annoying, papers that give such flawed cross validation results plus an independent test set that demonstrates the overoptimistic bias of the cross validation but not a single word that the design of the cross validation is wrong ... 
(I'd be perfectly happy if the same data would be presented "we know the cross validation should split patients, but we're stuck with software that doesn't allow this. Therefore we tested a truly independent set of test patients in addition")
(I'm also aware that bootstrapping = resampling with replacement usually performs better than cross validation = resampling without replacement. However, we found for spectroscopic data (simulated spectra and slightly artificial model setup but real spectra) that repeated/iterated cross validation and out-of-bootstrap had similar overall uncertainty; oob had more bias but less variance - for rewieving, I'm looking at this from a very pragmatic perspective: repeated cross validation vs. out-of-bootstrap does not matter as long as many papers neither split patient-wise nor report/discuss/mention random uncertainty due to limited test sample size.) 
Besides being wrong this also has the side effect that people who do a proper validation often have to defend why their results are so much worse than all those other results in the literature.
A: Using Microsoft Word rather than LaTeX.
A: Irene Stratton and colleague published a short paper about a closely related question:
Stratton IM, Neil A. How to ensure your paper is rejected by the statistical reviewer. Diabetic Medicine 2005; 22(4):371-373. 
A: Using "data" in a singular sense. Data ARE, they never is.
A: The code used to generate the simulated results is not provided. After asking for the code, it demands additional work to get it to run on a referee generated dataset.
A: Plagiarism (theoretical or methodological). My first review was indeed for a paper figuring many unreferenced copy/paste from a well-established methodological paper published 10 years ago.
Just found a couple of interesting papers on this topic: Authorship and plagiarism in science.
In the same vein, I find falsification (of data or results) the worst of all.
A: For me by far is , attributing cause without any proper causal analysis or when there is improper causal inference. 
I also hate it when zero attention is given to how missing data was handled.  I see so many papers too where the authors simply perform complete case analysis and make no mention of whether or not the results are generalizable to the population with missing values or how the population with missing values might be systematically different from the population with complete data.
A: When we ask the authors for 


*

*minor comment about an idea we have (in this sense, this not considered as a reason for rejecting the paper but just to be sure the authors are able to discuss another POV), or 

*unclear or contradicting results,


and that authors don't really answer in case (1) or that the incriminated results in (2) disappear from the MS.
A: Confusing p-values and effect size (i.e. stating my effect is large because I have a really tiny p-value).
Slightly different than Stephan's answer of excluding effect sizes but giving p-values. I agree you should give both (and hopefully understand the difference!)
A: Not including effect sizes.
P-ing all over the research (I have to credit my favorite grad school professor for that line).
Giving a preposterous number of digits (males gained 3.102019 pounds more than females)
Not including page numbers (that makes it harder to review)
Misnumbering figures and tables
(as already mentioned - stepwise and categorizing continuous variables)
A: When they don't sufficiently explain their analysis and/or include simple errors that make it difficult to work out what actually was done. This often includes throwing around a lot of jargon, by way of explanation, which is more ambiguous than the author seems to realize and also may be misused.
A: Using causal language to describe associations in observational data when omitted variables are almost certainly a serious concern.
A: Coming up with new words for the existing concepts, or, vice versa, using the existing terms to denote something different. 
Some of the existing terminology differentials has long settled in the literature: longitudinal data in biostatistics vs. panel data in econometrics; cause and effect indicators in sociology vs. formative and reflective indicators in psychology; etc. I still hate them, but at least you can find a few thousand references to each of them in their respective literatures. The most recent one is this whole strand of work on directed acyclic graphs in causal literature: most, if not all, of the theory of identification and estimation in these has been developed by econometricians in the 1950s under the name of simultaneous equations.
The term that has double, if not triple, meaning, is "robust", and the different meanings are often contradictory. "Robust" standard errors are not robust for far outliers; moreover, they are not robust to against anything except the assumed deviation from the model, and often have dismal small sample performance. White's standard errors are not robust against serial or cluster correlations; "robust" standard errors in SEM are not robust against the misspecifications of the model structure (omitted paths or variables). Just like with the idea of the null hypothesis significance testing, it is impossible to point a finger at anybody and say: "You are responsible for confusing several generations of researchers for coining this concept that does not really stand for its name".
A: When authors use the one statistical test they know (in my field, usually a t-test or an ANOVA), ad infinitum, regardless of whether it's appropriate.  I recently reviewed a paper where the authors wanted to compare a dozen different treatment groups, so they had done a two-sample t-test for every possible pair of treatments... 
A: Zero consideration of missing data. 
Many practical applications use data for which there are at least some missing values. This is certainly very true in epidemiology. Missing data presents problems for many statistical methods - including linear models. Missing data with linear models is often dealt with through deletion of cases with any missing data on any covariates. This is a problem, unless data are missing under an assumption that data are Missing Completely At Random (MCAR). 
Perhaps 10 years ago, it was reasonable to publish results from linear models with no further consideration of missingness.  I am certainly guilty of this. However, very good advice on how to deal with missing data with multiple imputation is now widely available, as are statistical packages/models/libraries/etc. to facilitate more appropriate analyses under more reasonable assumptions when missingness is present. 
