# How to annoy a statistical referee?

I recently asked a question regarding general principles around reviewing statistics in papers. What I would now like to ask, is what particularly irritates you when reviewing a paper, i.e. what's the best way to really annoy a statistical referee!

-
Does it extend to justifications received in response to an initial review (where minor and/or major revisions were asked)? –  chl Oct 20 '10 at 19:14
@chl: Yes, why not. –  csgillespie Oct 20 '10 at 19:19

What particularly irritates me personally is people who clearly used user-written packages for statistical software but don't cite them properly, or at all, thereby failing to give any credit to the authors. Doing so is particularly important when the authors are in academia and their jobs depend on publishing papers that get cited. (Perhaps I should add that, in my field, many of the culprits are not statisticians.)

-
+1 for me. This frustrates me, especially when they cite the wrong thing and I've provided the relevant details on how to cite the packages –  Gavin Simpson Oct 20 '10 at 19:36
Question: when citing a package, do you cite the vignette (if one exists) or the package itself? –  Brandon Bertelsen Nov 24 '10 at 6:55
@Brandon: if the package author cares enough to guide you, then they have given the answer in a form that will be picked up by citation("some_package") –  Ben Bolker Dec 14 '10 at 15:38

Goodness me, so many things come to mind...

• Stepwise regression

• Splitting continuous data into groups

• Giving p-values but no measure of effect size

• Describing data using the mean and the standard deviation without indicating whether the data were more-or-less symmetric and unimodal

• Figures without clear captions (are those error bars standard errors of the mean, or standard deviations within groups, or what?)

-
I'm a little curious about the stepwise regression bullet. What makes stepwise regression so bad? Is it the data dredging and multiple comparisons issue? –  Christopher Aden Oct 20 '10 at 19:56
The problem is that stepwise procedures completely invalidate all the assumptions and preconditions for "normal" inferential statistics based on p values, which are then badly biased (downwards towards being "more significant"). So basically, the answer is "yes", with the caveat that one could in principle correct for all these multiple comparisons (but which I have never seen done). I believe strongly that this is the single most important reason why I see so much research in psychology that cannot be replicated - which in turn leads to a huge waste of resources. –  Stephan Kolassa Oct 20 '10 at 20:04
@Stephan: I agree, stepwise is a bad idea. Though, while they may have not made it to psych methods yet, but there are a variety of selection procedures that adjust for bias related to overfitting by adjusting estimates and standard errors. This is not typically thought of as an issue of multiple comparisons. They are known as shrinkage methods. See my response in this thread <stats.stackexchange.com/questions/499/…; and Harrell's "Regression Modeling Strategies" or Tibshirani on the lasso. –  Brett Magill Oct 20 '10 at 20:34
@Brett Magill: +1 on that, and yes, I know about shrinkage and the lasso. Now all I need is some way to convince psychologists that these make sense... but people have been fighting with very limited success just to get psychologists to report confidence intervals, so I'm not too optimistic about psychologists' accepting shrinkage in the next twenty years. –  Stephan Kolassa Oct 20 '10 at 20:42
I'd also argue that in psychology maximising prediction is not typically the theoretical aim, yet stepwise regression is all about maximising prediction, albeit in a quasi-parsimonious way. Thus, there is typically a disconnect between procedure and question. –  Jeromy Anglim Oct 21 '10 at 6:28

Irene Stratton and colleague published a short paper about a closely related question:

Stratton IM, Neil A. How to ensure your paper is rejected by the statistical reviewer. Diabetic Medicine 2005; 22(4):371-373.

-

The code used to generate the simulated results is not provided. After asking for the code, it demands additional work to get it to run on a referee generated dataset.

-
And it's poorly formatted, uncommented, and uses indecipherable variable and function names. Ooooh yeah. –  naught101 Apr 18 '12 at 10:26

Plagiarism (theoretical or methodological). My first review was indeed for a paper figuring many unreferenced copy/paste from a well-established methodological paper published 10 years ago.

Just found a couple of interesting papers on this topic: Authorship and plagiarism in science.

In the same vein, I find falsification (of data or results) the worst of all.

-
Reminds me that in my early days as a referee i spent far too long reviewing a statistical paper that was eventually rejected by that particular journal, but the other referees and I suggested a more useful application for the method, and I also sketched an algebraic proof to replace an unsatisfactory simulation study in the manuscript. The authors have since got two published papers out of it. I'm not annoyed by that, but an acknowledgement such as "we thank referees of an earlier version of the paper for helpful comments" would have been good manners. –  onestop Oct 20 '10 at 21:23
@onestop Yes, I can imagine how disappointing such a situation might be... –  chl Oct 20 '10 at 21:56
@onestop, I hear you brother :-) –  suncoolsu Oct 21 '10 at 3:51
A few weeks ago I was given a paper to review and found that 85% of it had been published in another journal...by the same authors. That, too, is still considered plagiarism. For the last several years I have routinely submitted chunks of papers--especially abstracts, introductions, and conclusions--to Web search engines before doing any review. I want to be sure the work is original before I invest any time in reading it. –  whuber Oct 21 '10 at 4:40
+1, @whuber. As an editor of a methodological journal, I often have this tough job of figuring out whether the contribution (as a rule, from well established authors; the younger authors have not all gotten to that trajectory yet) warrants the publication given that all they've done is they reassembled in a different way the eight Lego blocks that comprised their previous five papers. This leads me to question the contribution in the preceding fifty papers these authors published, too :(. –  StasK Apr 10 '12 at 16:46

When we ask the authors for

1. minor comment about an idea we have (in this sense, this not considered as a reason for rejecting the paper but just to be sure the authors are able to discuss another POV), or

and that authors don't really answer in case (1) or that the incriminated results in (2) disappear from the MS.

-
Mysteriously disappearing results should be automatic rejection, imo. I'm sure this happens a lot "behind the scenes" (i.e. before the paper is submitted), but this is clear evidence of "cherry picking" that normal readers of the paper would never know. –  Macro Apr 13 '12 at 13:10
Another reason for an open peer review system. –  fmark Mar 3 '14 at 9:12

Confusing p-values and effect size (i.e. stating my effect is large because I have a really tiny p-value).

Slightly different than Stephan's answer of excluding effect sizes but giving p-values. I agree you should give both (and hopefully understand the difference!)

-

Not including effect sizes.

P-ing all over the research (I have to credit my favorite grad school professor for that line).

Giving a preposterous number of digits (males gained 3.102019 pounds more than females)

Not including page numbers (that makes it harder to review)

Misnumbering figures and tables

(as already mentioned - stepwise and categorizing continuous variables)

-
(+1) laughed out loud at "Giving a preposterous number of digits (males gained 3.102019 pounds more than females)". –  Macro Apr 13 '12 at 13:04

When they don't sufficiently explain their analysis and/or include simple errors that make it difficult to work out what actually was done. This often includes throwing around a lot of jargon, by way of explanation, which is more ambiguous than the author seems to realize and also may be misused.

-
Agree -- struggling to understand what the author(s) meant before even evaluating the scientific content is really annoying. –  Laurent Oct 22 '10 at 17:04
I agree but I find it even more annoying when a reviewer tells you to omit (or move to suppl. materials) what are, realistically, very crucial details about the analysis. This problem makes it so that lots of science/social science papers that do even the most slightly complicated analysis are pretty cryptic in that regard. –  Macro Apr 13 '12 at 13:08

When authors use the one statistical test they know (in my field, usually a t-test or an ANOVA), ad infinitum, regardless of whether it's appropriate. I recently reviewed a paper where the authors wanted to compare a dozen different treatment groups, so they had done a two-sample t-test for every possible pair of treatments...

-

Using causal language to describe associations in observational data when omitted variables are almost certainly a serious concern.

-
I agree that researchers should understand the liabilities of observational research designs, especially those related to omitted variables, but I don't think avoiding causal language does this. See the work of Hubert Blalock, in particular his book Causal Inferences in Non-experimental Research for a more detailed argument in defense of using causal language. –  Andy W May 22 '11 at 21:07
(+1) This might be my single biggest problem with epidemiological research. –  Macro Apr 13 '12 at 13:06

Coming up with new words for the existing concepts, or, vice versa, using the existing terms to denote something different.

Some of the existing terminology differentials has long settled in the literature: longitudinal data in biostatistics vs. panel data in econometrics; cause and effect indicators in sociology vs. formative and reflective indicators in psychology; etc. I still hate them, but at least you can find a few thousand references to each of them in their respective literatures. The most recent one is this whole strand of work on directed acyclic graphs in causal literature: most, if not all, of the theory of identification and estimation in these has been developed by econometricians in the 1950s under the name of simultaneous equations.

The term that has double, if not triple, meaning, is "robust", and the different meanings are often contradictory. "Robust" standard errors are not robust for far outliers; moreover, they are not robust to against anything except the assumed deviation from the model, and often have dismal small sample performance. White's standard errors are not robust against serial or cluster correlations; "robust" standard errors in SEM are not robust against the misspecifications of the model structure (omitted paths or variables). Just like with the idea of the null hypothesis significance testing, it is impossible to point a finger at anybody and say: "You are responsible for confusing several generations of researchers for coining this concept that does not really stand for its name".

-
I have to admit committing both sins: I describe my data as "having a hierarchical structure: when I have levels with 1 : n relations (many measurements of each sample, multiple samples per patient). At some point I rather accidentally learned that this is called a "clustered" data structure - now I use both terms. But I still don't know how I could have found that term, I did look desparately for the word to describe my data structure... The other way round: I use techniques that are called soft classification in remote sensing. My field (chemometrics) uses it with quite different meaning. –  cbeleites Apr 10 '12 at 17:29
That's all fine -- you can add "multilevel" to your list of the ways to refer to this structure, too. "Clustered" usually means that the observations are known to be correlated, but nobody cares to model that correlation since it is not of primary interest, and gets away methods that are robust to such correlation, such as GEE. What you have is something like repeated measures MANOVA. There's a Stata package gllamm that thinks about your data as a multilevel/hierarchical data, but most other packages would think of multiple measurements as variables/columns, and samples as observations/rows. –  StasK Apr 11 '12 at 9:09
Thanks for the input. Well, nowadays I'd of course ask here how it is called... It's not exactly repeated measurements: usually I measure a number (order of magnitude: between 10^2 and 10^4) different spots on the sample in order to produce false-colour maps of different constituents, and each measurement already has 10^2 - 10^3 observations (wavelengths in the spectrum). Within each sample, many spectra are highly correlated, but not all: the samples are not homogeneous. ... –  cbeleites Apr 11 '12 at 15:47
... Your description of "clustered" sounds very much like what we do. But I do take care to split the samples for validation, say I don't have any idea about the effective sample size (besides that it is at least the number of real samples involved), and sometimes show that having all those measurements of each sample actually helps for the model training. –  cbeleites Apr 11 '12 at 15:49
Interesting and challenging data, for sure. –  StasK Apr 11 '12 at 17:13

Zero consideration of missing data.

Many practical applications use data for which there are at least some missing values. This is certainly very true in epidemiology. Missing data presents problems for many statistical methods - including linear models. Missing data with linear models is often dealt with through deletion of cases with any missing data on any covariates. This is a problem, unless data are missing under an assumption that data are Missing Completely At Random (MCAR).

Perhaps 10 years ago, it was reasonable to publish results from linear models with no further consideration of missingness. I am certainly guilty of this. However, very good advice on how to deal with missing data with multiple imputation is now widely available, as are statistical packages/models/libraries/etc. to facilitate more appropriate analyses under more reasonable assumptions when missingness is present.

-
In the spirit of trying to educate, can you elaboorate more? What do you consider consideration - admitting it exists or adjusting the statistical analysis in the face of it (e.g. imputation). When applicable I try to include supp. tables of missing values by covariates of interest, but it isn't clear if this is sufficient for "consideration" by this remark. –  Andy W Apr 3 '13 at 16:04
Fair enough.... –  D L Dahly Apr 3 '13 at 16:20
Thank you for the update! –  Andy W Apr 3 '13 at 16:57

I'm most (and most frequently) annoyed by "validation" aiming at generalization error of predictive models where the test data is not independent (e.g. typically multiple measurements per patient in the data, out-of-bootstrap or cross validation splitting measurements not patients).

Even more annoying, papers that give such flawed cross validation results plus an independent test set that demonstrates the overoptimistic bias of the cross validation but not a single word that the design of the cross validation is wrong ...

(I'd be perfectly happy if the same data would be presented "we know the cross validation should split patients, but we're stuck with software that doesn't allow this. Therefore we tested a truly independent set of test patients in addition")

(I'm also aware that bootstrapping = resampling with replacement usually performs better than cross validation = resampling without replacement. However, we found for spectroscopic data (simulated spectra and slightly artificial model setup but real spectra) that repeated/iterated cross validation and out-of-bootstrap had similar overall uncertainty; oob had more bias but less variance - for rewieving, I'm looking at this from a very pragmatic perspective: repeated cross validation vs. out-of-bootstrap does not matter as long as many papers neither split patient-wise nor report/discuss/mention random uncertainty due to limited test sample size.)

Besides being wrong this also has the side effect that people who do a proper validation often have to defend why their results are so much worse than all those other results in the literature.

-
Not sure if you meant to say this but the "optimism" bootstrap is one of the best ways to validate a model, and its training and test samples overlap. –  Frank Harrell Apr 12 '12 at 3:00
@Frank Harrell - I'm not sure I got your point. Maybe the difficulty is that in chemometrics "validation of a predictive model" always is about performance for new, unknown, future cases (in the example: diagnosing new patients). I use out-of-bootstrap or repeated/iterated cross validation all the time. Can you explain what the advantage of having test & train sets overlapping is compared to splitting at the patient level (I assume "overlapping" means splitting measurements so test & training measurements can belong to the same patient, always talking about an inter-patient model)? –  cbeleites Apr 12 '12 at 15:40
... And yes, some points of the model validation can be answerd without splitting the data in distinct test and training cases (e.g. model stability in terms of the coefficients). But already model stability wrt. to the predictions should be measured using unknown patients (unknown: never appeared in the process of building the model including any data-driven pre-processing that takes all cases into account). Actually, for a traditional quantitation in chemometrics, the validation has steps that need further independently measured test data: ... –  cbeleites Apr 12 '12 at 15:48
good practice calls for unknown operator of the instrument and one important characteristic of the analytical method to be determined during validation is how often the calibration needs to be re-done (or showing that instrumental drift is negligible over a certain amount of time) - some authors even talk about an "abuse of resampling" that leads to a neglect of such independent test sets. –  cbeleites Apr 12 '12 at 15:53
If the equipment or measurement techniques are in need of validation, then an independent sample is required. But a common mistake is to use data splitting to try to simulate an independent validation. This is still an internal validation. To answer @cbeleites question above, the overlapped samples involved with bootstrapping will result in more accurate estimates of future model performance than data splitting in the majority of datasets one is likely to see. I have had data splitting perform poorly with n=17,000 and 0.30 event rate. –  Frank Harrell Apr 12 '12 at 21:44

Reporting effects that "approached significance ( p < .10 for example) and then writing about them as though they had attained significance at a more stringent and acceptable level. Running multiple Structural Equation Models that were not nested and then writing about them as though they were nested. Taking a well-established analytic strategy and presenting it as though no-one had ever thought of using it before. Perhaps this qualifies as plagiarism to the nth degree.

-
Maybe it's reinventing the wheel rather than plagiarism? –  gerrit Jun 11 at 13:36

I recommend the following two articles:

Martin Bland:
How to Upset the Statistical Referee
This is based on a series of talks given by Martin Bland, along with data from other statistical referees (‘a convenience sample with a low response rate’). It ends with an 11-point list of ‘[h]ow to avoid upsetting the statistical referee’.

Stian Lydersen: