33
$\begingroup$

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply. My specific wording is usually given in a footnote:

"While, strictly speaking, inferential statistics are only applicable in the context of random sampling, we follow convention in reporting significance levels and/or confidence intervals as convenient yardsticks even for nonrandom samples. See Michael Oakes's Statistical inference: A commentary for the social and behavioural sciences (NY: Wiley, 1986).

On a couple of occasions--once for a peer-reviewed paper, once or twice in a non-academic setting--the editor or reviewer objected to this disclaimer, calling it confusing, and felt that the inferential findings should simply stand as written (and be given the mantle of authority). Has anyone else encountered this problem and found a good solution? On the one hand, people's understanding of p-values is generally dismal, even in the context of random sampling, so perhaps it doesn't matter much what we say. On the other, to contribute further to misunderstandings seems to make one part of the problem. I should add that I frequently deal with survey studies, where random assignment does not apply and where Monte Carlo simulations would often fail to address the issue of representativeness.

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ the comment on a reviewer is extremely sad, one would hope that a person in that position would at least not openly display their ignorance, and by doing so, further support the misinterpretation of the statistical method. $\endgroup$
    – richiemorrisroe
    Feb 12, 2011 at 17:29
  • 8
    $\begingroup$ Correct me if I'm wrong, but the randomness of sampling simply affects the degree to which you can generalize findings. In contrast, random assignment is the more critical feature for causal inference. $\endgroup$
    – Mike Lawrence
    Feb 12, 2011 at 17:29
  • 3
    $\begingroup$ Mike, I agree with you. Do you make this point to extend the discussion or to indicate disagreement with something I've said? $\endgroup$
    – rolando2
    Feb 12, 2011 at 23:08
  • $\begingroup$ @richiemorrisroe: one would be foolish to expect that of all reviewers, but I suppose one can hope for a future in which we can expect that, and we should certainly pressure publishers to do more to demand and enforce that than they do presently...Rolando, I think Mike's is merely a point of clarification to disambiguate this discussion from causal-inference-related issues. Evidently some people have found that helpful, though I thought it was clear enough already, personally. If I'm right, this inadvertently measures others' confusion about p values, which motivates the original post! $\endgroup$
    – Nick Stauner
    Nov 24, 2013 at 12:28

3 Answers 3

Reset to default
11
$\begingroup$

There is indeed an argument to be had not to include the disclaimer. Frankly, I'd find a brief treatise on the nature of p-values in a journal article to be a little off-putting, and for a moment would have to pause and try to figure out if you'd done something particularly...esoteric...to warrant devoting that space to a definitional point.

Basically, as a reviewer, I'd call it unnecessary because the reader should already know what a p-value is and does. I might even object to it because making such a note does not actually prevent any of the many crimes of analysis and interpretation that accompany p-values, it merely puts on a cloak of "trust me, I know what I'm doing". It's also a little odd - "I'm going to make a bold stand against p-values, but not so bold I don't report them".

When I consider "entrenched views on p-values", I'm much less concerned about something like what you posted above, and much more concerned about reviewers' insistence on statistical significance in order to be published or the focus of the paper (put a star by a finding and suddenly its a Big Deal) or blending statistical significance with the significance of a finding.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ I don't think this answers the OP. I'm assuming @rolando2 reports other stats that are more central to his discussions (e.g., effect sizes), and mostly reports p values as one way of accommodating conventional expectations, even though they don't apply strictly. As such, we should be off-put to whatever extent we're reading too much into p values; we should consider his motivation for the disclaimer. Readers don't know what they should; the OP mentions this. The disclaimer promotes doubt, not trust. It's not that odd to object to a standard while conforming to it; it's not a bold stand. $\endgroup$
    – Nick Stauner
    Nov 24, 2013 at 12:48
  • $\begingroup$ @NickStauner I don't see how it doesn't "answer" the OP. Perhaps it doesn't support what they want to do, but in my mind it's both a really weird break from the actual content of the paper, and also useless - "This is wrong, but I'm going to soldier on as if its right because it's what you all expect" doesn't tell me if the wrongness matters. $\endgroup$
    – Fomite
    Dec 3, 2013 at 22:05
  • 3
    $\begingroup$ The OP's question: "Has anyone else encountered this problem and found a good solution?" Your answer ignores the literal question to respond to the idea, and mostly offers your opinions on why the idea should be shot down. You're starting to hint at a constructive critique of the OP's idea though: you don't seem to think the Oakes citation tells you why it matters. I'll expand on this a little in an answer of my own. $\endgroup$
    – Nick Stauner
    Dec 10, 2013 at 2:04
13
$\begingroup$

The use of inferential statistics can be justified not only based on a population model, but also based on a randomization model. The latter does not make any assumptions about the way the sample has been obtained. In fact, Fisher was the one that suggested that the randomization model should be the basis for statistical inference (as opposed to Neyman and Pearson). See, for example:

Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19, 676-685. [link (open access)]

Ludbrook, J. & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52, 127-132. [link (if you have JSTOR access)]

I somehow doubt though that the editors or reviewers in question were using this as the reason for calling your disclaimer "confusing".

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Wolfgang - interesting and helpful points. I should have made clear, though, that much of my work is on surveys. $\endgroup$
    – rolando2
    Feb 12, 2011 at 23:28
  • 7
    $\begingroup$ If the primary goal is to make some kind of inference to the population and the sampling mechanism is of such a nature that the representativeness of the sample is questionable, then indeed, any inference will also be rather questionable. Essentially, you can only make an inference to that part of the population that the sampling mechanism provides a representation of. In principle, the inferences you make will be appropriate for that part of the population. Whether that part of the population is of any interest to you (or the readers) is another issue. $\endgroup$
    – Wolfgang
    Feb 15, 2011 at 10:24
7
$\begingroup$

I haven't had to do battle with any bad reviewers yet, so I wouldn't claim any knowledge of how to get out of a battle that's already begun. However, if their objections are a mere matter of obstructive ignorance, a little preemptive diversion might do the trick. If $p$ values are in fact necessary to report despite their non-negligible invalidity in a problematic study (a class into which all too many published articles fall), one might downplay them implicitly. Consider focusing your narrative instead—maybe even exclusively—on effect sizes. If your study is sufficiently representative to be usefully informative (this shouldn't necessitate perfectly random sampling, only caution in the generality of interpretations), your effect sizes ought to have broader implications than merely indicating the existence and directions of relationships or differences anyway. Focusing one's discussion on effect sizes can facilitate a deeper understanding of how much the relationships or differences matter in a practical sense, though this still needs to be considered in the context of the subject of study (e.g., one cannot conclude by size alone that an $r = .03$ is necessarily unimportant if it might pertain to a matter of life and death; Rosenthal, Rubin, & Rosnow, 2000). You can do this by discussing results in terms of "weak," "moderate," or "strong" relationships or "small" or "large" differences instead of referring to them as "significant" and "insignificant"; the latter two words shouldn't be necessary whatsoever to make most of the points researchers want to make. If the $p$ values are necessary, let them speak for themselves. Do meta-analysts a favor and just sandwich them in more comprehensive reports of valuable statistics: effect sizes, confidence intervals, and test statistics. Maybe hope for a day when readers and reviewers will ignore $p$ values and demand confidence intervals, so that the $p$ values can be ditched entirely. (Or maybe not! See post-postscript!)

Another, potentially complementary option would be to expand on your footnote. Both your descriptions of the problem as reviewers have experienced it, and the presently accepted answer on this page, suggest that not enough information is conveyed to explain your motivation for including the footnote, nor enough to motivate the reader to follow your citation to the reference that you use to explain it so tersely. A single, additional sentence, even a brief quote from your reference, could go a long way toward explaining the value of your footnote and motivating readers to read deeper. Evidently, your footnote as is sooner motivates a simple, negative, dismissive reaction toward your understated attempt to disrupt their complacency about their improper assumptions. Readers might be a little less intellectually lazy if you spoonfeed them one or two of the main points about problems that they probably overlook routinely. Also, for many particular problems with $p$ values, consider citing not just that book, but also a fairly concise journal article that's freely available online presently (e.g., Goodman, 2008, Wagenmakers, 2007). That might help reduce any resistance due to the difficulty of obtaining a book and finding the relevant info within.

P.S. Thanks to @rpierce for Wagenmakers (2007) and much of the logic of my answer, and to @FranciscoArceo for Goodman (2008)! See also Francisco's loosely related answer, as well as some other popular posts here on Cross Validated about interpreting $p$ values properly:

P.P.S. @MichaelLew's counterpoint is also worth considering before tossing the $p$ values out entirely! See Senn (2001) and Lew (2013) for some rare and valuable (but only partial) defenses of $p$. [Edit]: Also, I brought up this question in a new question, "Why are 0.05 < p < 0.95 results called false positives?" In discussing my answer, the OP brought up Hurlbert and Lombardi (2009), which I brought up with my colleagues, one of whom then brought up Nuzzo (2014), a brand new Nature News article that led to even more references (Goodman, 2001, 1992; Gorroochurn, Hodge, Heiman, Durner, & Greenberg, 2007)...I am obviously not keeping up at this point, but Michael is just as clearly not alone in defending the possibility of extracting useful information from exact $p$ values (when they do "strictly apply", at least).

References

- Goodman, S. N. (1992). A comment on replication, P‐values and evidence. Statistics in Medicine, 11(7), 875–879.
- Goodman, S. N. (2001). Of P-values and Bayes: A modest proposal. Epidemiology, 12(3), 295–297. Retrieved from http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/ETP_Cetacean_Assessment/Of_P_Values_and_Bayes__A_Modest_Proposal.6.pdf.
- Goodman, S. (2008). A dirty dozen: Twelve P-value misconceptions. Seminars in Hematology, 45(3), 135–140. Retrieved from http://xa.yimg.com/kq/groups/18751725/636586767/name/twelve+P+value+misconceptions.pdf.
- Gorroochurn, P., Hodge, S. E., Heiman, G. A., Durner, M., & Greenberg, D. A. (2007). Non-replication of association studies: “pseudo-failures” to replicate? Genetics in Medicine, 9(6), 325–331. Retrieved from http://www.nature.com/gim/journal/v9/n6/full/gim200755a.html.
- Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman–Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. Retrieved from http://xa.yimg.com/kq/groups/1542294/508917937/name/HurlbertLombardi2009AZF.pdf.
- Lew, M. J. (2013). To P or not to P: On the evidential nature of P-values and their place in scientific inference. arXiv:1311.0081 [stat.ME]. Retrieved from http://arxiv.org/abs/1311.0081.
- Nuzzo, R. (2014, February 12). Scientific method: Statistical errors. Nature News, 506(7487). Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700.
- Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge University Press.
- Senn, S. (2001). Two cheers for P-values? Journal of Epidemiology and Biostatistics, 6(2), 193–204. Retrieved from http://www.phil.vt.edu/dmayo/conference_2010/Senn%20Two%20Cheers%20Paper.pdf.
- Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. Retrieved from http://www.brainlife.org/reprint/2007/Wagenmakers_EJ071000.pdf.

Improve this answer
$\endgroup$
13
  • 3
    $\begingroup$ Not all of the criticisms of P-values are correct or warranted, despite their vehemence. You should see these two papers for a few counterpoints to the commentaries that you cite: Two Cheers for P Values (by Stephen Senn) phil.vt.edu/dmayo/conference_2010/…; To P or Not To P (by me) arxiv.org/abs/1311.0081 $\endgroup$
    – Michael Lew
    Dec 10, 2013 at 5:15
  • 1
    $\begingroup$ Excellent point! Thank you! I've edited slightly to include your contributions, and I may edit a bit more once I understand them well enough to incorporate their implications into the rest of what I've said. This is why I love Cross Validated... $\endgroup$
    – Nick Stauner
    Dec 10, 2013 at 12:21
  • 2
    $\begingroup$ Do you have any experimental evidence in favor of your claim that Edwards' assumption extends to p-values? I find my self extremely skeptical. My way of analogy, I've seen a couple papers that demonstrate that even experienced data scientists have trouble estimating a correlation coefficient from a scatterplot. It seems like you are asking a lot more from scientists in gaining a sense of what a p value means in terms of likelihood. Your argument in favor of likelihood functions is interesting... they do tend to look a bit like posterior distributions, no? $\endgroup$
    – russellpierce
    Feb 7, 2014 at 5:46
  • 2
    $\begingroup$ @rpierce I don't have experimental evidence for the understanding of users of statistical methods. I would, however, contend that at least some of the studies that have been done to see if scientists 'understand' p-values are fatally flawed by not including among the options a true evidentially meaningful description of the p-value. Your analogy is not close because the fact that correlation coefficients are not easily estimated is not the same problem as estimating the strength of evidence from a p-value. $\endgroup$
    – Michael Lew
    Feb 9, 2014 at 20:06
  • 1
    $\begingroup$ @rpierce The posterior probability density function from a uniform prior will be proportional to the likelihood function. $\endgroup$
    – Michael Lew
    Feb 9, 2014 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.