Accommodating entrenched views of p-values

Question

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply. My specific wording is usually given in a footnote:

"While, strictly speaking, inferential statistics are only applicable in the context of random sampling, we follow convention in reporting significance levels and/or confidence intervals as convenient yardsticks even for nonrandom samples. See Michael Oakes's Statistical inference: A commentary for the social and behavioural sciences (NY: Wiley, 1986).

On a couple of occasions--once for a peer-reviewed paper, once or twice in a non-academic setting--the editor or reviewer objected to this disclaimer, calling it confusing, and felt that the inferential findings should simply stand as written (and be given the mantle of authority). Has anyone else encountered this problem and found a good solution? On the one hand, people's understanding of p-values is generally dismal, even in the context of random sampling, so perhaps it doesn't matter much what we say. On the other, to contribute further to misunderstandings seems to make one part of the problem. I should add that I frequently deal with survey studies, where random assignment does not apply and where Monte Carlo simulations would often fail to address the issue of representativeness.

Answer 1

There is indeed an argument to be had not to include the disclaimer. Frankly, I'd find a brief treatise on the nature of p-values in a journal article to be a little off-putting, and for a moment would have to pause and try to figure out if you'd done something particularly...esoteric...to warrant devoting that space to a definitional point.

Basically, as a reviewer, I'd call it unnecessary because the reader should already know what a p-value is and does. I might even object to it because making such a note does not actually prevent any of the many crimes of analysis and interpretation that accompany p-values, it merely puts on a cloak of "trust me, I know what I'm doing". It's also a little odd - "I'm going to make a bold stand against p-values, but not so bold I don't report them".

When I consider "entrenched views on p-values", I'm much less concerned about something like what you posted above, and much more concerned about reviewers insistence on statistical significance in order to be published or the focus of the paper (put a star by a finding and suddenly its a Big Deal) or blending statistical significance with the significance of a finding.

Answer 2

The use of inferential statistics can be justified not only based on a population model, but also based on a randomization model. The latter does not make any assumptions about the way the sample has been obtained. In fact, Fisher was the one that suggested that the randomization model should be the basis for statistical inference (as opposed to Neyman and Pearson). See, for example:

Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19, 676-685. [link (open access)]

Ludbrook, J. & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52, 127-132. [link (if you have JSTOR access)]

I somehow doubt though that the editors or reviewers in question were using this as the reason for calling your disclaimer "confusing".