Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Edit: The basis of my question is flawed, and I need to spend some time figuring out whether it can even be made to make sense.

Edit 2: Clarifying that I recognize that a p-value isn't a direct measure of the probability of a null hypothesis, but that I assume that the closer a p-value is to 1, the more likely it is that a hypothesis has been chosen for experimental testing whose corresponding null hypothesis is true, while the closer a p-value is to 0, the more likely it is that a hypothesis has been chosen for experimental testing whose corresponding null hypothesis is false. I can't see how this is false unless the set of all hypotheses (or all hypotheses picked for experiments) is somehow pathological.

Edit 3: I think I'm still not using clear terminology to ask my question. As lottery numbers are read out, and you match them to your ticket one-by-one, something changes. The probability that you have won does not change, but the probability that you can turn the radio off does. There's a similar change that happens when experiments are done, but I have a feeling that the terminology I'm using - "p-values change the likelihood that a true hypothesis has been chosen" - isn't the correct terminology.

Edit 4: I've received two amazingly detailed and informative answers that contain a wealth of information for me to work through. I'll vote them both up now and then come back to accept one when I've learned enough from both answers to know that they've either answered or invalidated my question. This question opened a much bigger can of worms than the one I was expecting to eat.

In papers I've read, I've seen results with p > 0.05 after validation called "false positives". However, isn't it still more likely than not that I've chosen a hypothesis to test with a false corresponding null hypothesis when the experimental data has a p < 0.50 which is low but > 0.05, and aren't both the null hypothesis and the research hypothesis statistically uncertain/insignificant (given the conventional statistical significance cutoff) anywhere between 0.05 < p < 0.95 whatever the inverse of p < 0.05 is, given the asymmetry pointed out in @NickStauner's link?

Let's call that number A, and define it as the p-value which says the same thing about the likelihood that you've picked a true null hypothesis for your experiment/analysis that a p-value of 0.05 says about the likelihood that you've picked a true non-null hypothesis for your experiment/analysis. Doesn't 0.05 < p < A just say, "Your sample size wasn't big enough to answer the question, and you won't be able to judge application/real-world significance until you get a bigger sample and get your statistical significance sorted out"?

In other words, shouldn't it be correct to call a result definitely false (rather than simply unsupported) if and only if p > A?

This seems straightforward to me, but such widespread usage tells me that I might be wrong. Am I:

a) misinterpreting the mathematics,
b) complaining about a harmless-if-not-exactly-correct convention,
c) completely correct, or
d) other?

I recognize that this sounds like a call for opinions, but this seems like a question with a definite mathematically correct answer (once a significance cutoff is set) that either I or (almost) everybody else is getting wrong.

Thanks.

share|improve this question
1  
Hi David. Here's the paper that got me thinking about it: link –  Andrew Klaassen Feb 27 at 20:44
2  
In your first line don't you mean instead to write "... results originally with $p\lt 0.05$ but subsequently with $p\ge 0.05$ after validation..."? A result with $p$ greater than the threshold $\alpha$ otherwise is termed a negative result. Even after your edits, your characterization of the interpretation of $p$ is incorrect, so I would like to suggest you take a few moments to review some of our posts on interpreting p-values and reconsider what you want to ask. –  whuber Feb 27 at 21:04
1  
You can delete your question if you wish, but as you've received two upvotes (oh heck, let's make it 3), an upvoted answer, and are about to receive another answer from "yours truly", I request that you leave it active and work on it as you see fit, though I respectfully defer to your right to do as you wish. Cheers! –  Nick Stauner Feb 27 at 21:13
1  
I agree with @Nick, Andrew: you have a cogent and provocative question here that has attracted some thought and attention, so we would be most grateful if you would keep it posted and, if you can, refine it a little to focus on the key issue concerning how p-values are interpreted. The novel part, from what I can tell, is the suggestion that the rejection criterion should be based on a large p-value. Re your comment: a false positive occurs when the test is significant but it is known that the null hypothesis is true. –  whuber Feb 27 at 21:23
1  
@whuber: The more compelling background focus for me is on what result would suggest that a follow-up experiment with a larger sample size is likely to be productive. Given the responses so far, it seems I need to ask whether p-values could even be related to that question. Re knowing that the null hypothesis is true as the measure of a false positive: When would one say that a null hypothesis is true outside the situation p > (1 - α)? –  Andrew Klaassen Feb 27 at 21:45
show 11 more comments

2 Answers 2

up vote 10 down vote accepted

Your question is based on a false premise:

isn't the null hypothesis still more likely than not to be wrong when p < 0.50

A p-value is not a probability that the null hypothesis is true. For example, if you took a thousand cases where the null hypothesis is true, half of them will have p < .5. Those half will all be null.

Indeed, the idea that p > .95 means that the null hypothesis is "probably true" is equally misleading. If the null hypothesis is true, the probability that p > .95 is exactly the same as the probability that p < .05.

ETA: Your edit makes it clearer what the issue is: you still do have the issue above (that you're treating a p-value as a posterior probability, when it is not). It's important to note that this is not a subtle philosophical distinction (as I think you're implying with your discussion of the lottery tickets): it has enormous practical implications for any interpretation of p-values.

But there is a transformation you can perform on p-values that will get you to what you're looking for, and it's called the local false discovery rate. (As described by this nice paper, it's the frequentist equivalent of the "posterior error probability", so think of it that way if you like).

Let's work with a concrete example. Let's say you are performing a t-test to determine whether a sample of 10 numbers (from a normal distribution) has a mean of 0 (a one-sample, two-sided t-test). First, let's see what the p-value distribution looks like when the mean actually is zero, with a short R simulation:

null.pvals = replicate(10000, t.test(rnorm(10, mean=0, sd=1))$p.value)
hist(null.pvals)

enter image description here

As we can see, null p-values have a uniform distribution (equally likely at all points between 0 and 1). This is a necessary condition of p-values: indeed, it's precisely what p-values mean! (Given the null is true, there is a 5% chance it is less than .05, a 10% chance it is less than .1...)

Now let's consider the alternative hypothesis- cases where the null is false. Now, this is a bit more complicated: when the null is false, "how false" is it? The mean of the sample isn't 0, but is it .5? 1? 10? Does it randomly vary, sometimes small and sometimes large? For simplicity's sake, let's say it is always equal to .5 (but remember that complication, it'll be important later):

alt.pvals = replicate(10000, t.test(rnorm(10, mean=.5, sd=1))$p.value)
hist(alt.pvals)

enter image description here

Notice that the distribution is now not uniform: it is shifted towards 0! In your comment you mention an "asymmetry" that gives information: this is that asymmetry.

So imagine you knew both of those distributions, but you're working with a new experiment, and you also have a prior that there's a 50% chance it's null and 50% that it's alternative. You get a p-value of .7. How can you get from that and the p-value to a probability?

What you should do is compare densities:

lines(density(alt.pvals, bw=.02))
plot(density(null.pvals, bw=.02))

And look at your p-value:

abline(v=.7, col="red", lty=2)

enter image description here

That ratio between the null density and the alternative density can be used to calculate the local false discovery rate: the higher the null is relative to the alternative, the higher the local FDR. That's the probability that the hypothesis is null (technically it has a stricter frequentist interpretation, but we'll keep it simple here). If that value is very high, then you can make the interpretation "the null hypothesis is almost certainly true." Indeed, you can make a .05 and .95 threshold of the local FDR: this would have the properties you're looking for. (And since local FDR increases monotonically with p-value, at least if you're doing it right, these will translate to some thresholds A and B where you can say "between A and B we are unsure").

Now, I can already hear you asking "then why don't we use that instead of p-values?" Two reasons:

  1. You need to decide on a prior probability that the test is null
  2. You need to know the density under the alternative. This is very difficult to guess at, because you need to determine how large your effect sizes and variances can be, and how often they are so!

You do not need either of those for a p-value test, and a p-value test still lets you avoid false positives (which is its primary purpose). Now, it is possible to estimate both of those values in multiple hypothesis tests, when you have thousands of p-values (such as one test for each of thousands of genes: see this paper or this paperfor instance), but not when you're doing a single test.

Finally, you might say "Isn't the paper still wrong to say a replication that leads to a p-value above .05 is necessarily a false positive?" Well, while it's true that getting one p-value of .04 and another p-value of .06 doesn't really mean the original result was wrong, in practice it's a reasonable metric to pick. But in any case, you might be glad to know others have their doubts about it! The paper you refer to is somewhat controversial in statistics: this paper uses a different method and comes to a very different conclusion about the p-values from medical research, and then that study was criticized by some prominent Bayesians (and round and round it goes...). So while your question is based on some faulty presumptions about p-values, I think it does examine an interesting assumption on the part of the paper you cite.

share|improve this answer
    
Hi David. Fair point. I will work on re-phrasing my question to not get that part wrong, and see if I still see a problem. –  Andrew Klaassen Feb 27 at 20:54
    
@David_Robinson: Would it be correct to use the p-value as the false-alarm rate in the Bayesian rule, and be able to draw conclusions about the probability of the research and/or null hypotheses from that? Set the prior to 50% and play fast and loose from there? :-) –  Andrew Klaassen Feb 27 at 23:00
1  
Aye, fascinating! Can you work it into your answer? But there's an asymmetry between how p behaves when null is true versus when it's false that ~must~ give some information about the likelihood that the null hypothesis is true based on the p-value extracted from the data. If a true null hypothesis produces uniformly distributed p-values, and a true not-null hypothesis produces p-values that are skewed toward 0, pulling out a p=0.01 marble ~must~ suggest that you're more likely to have picked the not-null jar of experiments, even if the probability isn't changed by doing the experiment. –  Andrew Klaassen Feb 28 at 2:07
1  
@AndrewKlaassen: You may be interested in the concept of the "local false discovery rate". It's a frequentist equivalent to the Bayesian posterior probability that the null is true. It requires two things: a) A prior probability that the null is true (sometimes called pi0), and b) an estimate of the density for the alternative hypothesis. In multiple hypothesis testing (if you had thousands of p-values), it is possible to estimate both of these by looking at the density. If I have a bit more time I may build a deeper explanation into my answer. –  David Robinson Feb 28 at 4:32
1  
@AndrewKlaassen: See my edit, where I explain local FDR in detail, why it's the way to calculate your value "A" (though you might want to change .05 while you're calculating A), and also why it is rarely used. Anyway, to clarify one point that doesn't really fit in the answer: your example with the lottery ticket misunderstands the point that I and others have been making. We weren't getting hung up on the idea of "do probabilities change with new information" (both Bayesians and frequentists have their interpretation of that): the point is just that you weren't changing them the right way! –  David Robinson Feb 28 at 6:47
show 14 more comments

Hover your mouse over any ($\leftarrow$ is a fake tag) appearing below to see a brief excerpt of its wiki. Please forgive the disruption of line spacing. I find it worthwhile because tag excerpts may help readers to check understanding of jargon while reading through. Some of these excerpts may deserve editing as well, so they also deserve a publicist, IMHO.

$p>.05$ ordinarily implies one should not reject the . Conversely, or false positives occur when one does reject the null due to error or some other unusual incident that produces a that was otherwise unlikely (usually with $p<.05$) to have been sampled randomly from a in which the null is true. A result with $p>.05$ that is called a false positive seems to reflect a misunderstanding of null hypothesis ing (NHST). Misunderstandings are not uncommon in published research literature, as NHST is notoriously counter-intuitive. This is one of the rallying cries of the invasion (which I support, but do not follow...yet). I have worked with mistaken impressions such as these myself until recently, so I sympathize most heartily.

@DavidRobinson is correct in observing that $p$ is not the probability of the null being false in NHST. This is (at least) one of Goodman's (2008) "Dirty Dozen" misconceptions about $p$ values (see also Hurlbert & Lombardi, 2009). In NHST, $p$ is the that one would draw any future random samples by the same means that would exhibit a relationship or difference (or whatever is being tested against the null, if other varieties of effect size exist...?) at least as different from the null hypothesis as the sample(s) from the same population(s) one has tested to arrive at a given $p$ value, if the null is true. That is, $p$ is the probability of obtaining a sample like yours given the null; it does not reflect the probability of the null – at least, not directly. Conversely, Bayesian methods pride themselves on their formulation of statistical analyses as focused on estimating the evidence for or against a theory of an effect given the data, which they argue is a more intuitively appealing approach (Wagenmakers, 2007), among other advantages, and setting aside debatable disadvantages. (To be fair, see "What are the cons of Bayesian analysis?" You have also commented to cite articles that might offer some nice answers there: Moyé, 2008; Hurlbert & Lombardi, 2009.)

Arguably, the null hypothesis as literally stated is often more likely than not to be wrong, because null hypotheses are most commonly, literally hypotheses of zero effect. (For some handy counter-examples, see answers to: "Are large data sets inappropriate for hypothesis testing?") Philosophical issues such as the butterfly effect threaten the literal of any such hypothesis; hence the null is useful most generally as a basis of comparison for an alternative hypothesis of some nonzero effect. Such an alternative hypothesis may remain more plausible than the null after data have been collected that would've been improbable if the null were true. Hence researchers typically infer support for an alternative hypothesis from evidence against the null, but that is not what quantify directly (Wagenmakers, 2007).

As you suspect, is a function of , as well as effect size and consistency. (See @gung's answer to the recent question, "How can a t-test be statistically significant if the mean difference is almost 0?") The questions we often intend to ask of our data are, "What is the effect of x on y?" For various reasons (including, IMO, misconceived and otherwise deficient educational programs in statistics, especially as taught by non-statisticians), we often find ourselves instead asking literally the loosely related question, "What is the probability of sampling data such as mine randomly from a population in which x does not affect y?" This is the essential difference between effect size estimation and significance testing, respectively. A $p$ value answers only the latter question directly, but several professionals (@rpierce could probably give you a better list than I; forgive me for dragging you into this!) have argued that researchers misread $p$ as an answer to the former question of effect size all too often; I'm afraid I must agree.

To respond more directly regarding the meaning of $.05<p<.95$, it is that the probability of sampling data randomly from a population of which the null is true, but that exhibits a relationship or difference that differs from that which the null describes literally by at least as wide and consistent a margin as your data does...< inhale>...is between 5–95%. One may certainly argue this is a consequence of sample size, because increasing sample size improves one's ability to detect small and inconsistent effect sizes and differentiate them from a null of, say, zero effect with confidence exceeding 5%. However, small and inconsistent effect sizes may or may not be significant pragmatically ( $\ne$ significant statistically – another of Goodman's (2008) dirty dozen); this depends far more on the meaning of the data, with which statistical significance only concerns itself to a limited extent. See my answer to the above.

Shouldn't it be correct to call a result definitely false (rather than simply unsupported) if...p > 0.95?

Since data should usually represent empirically factual observations, they should not be false; only inferences about them should face this risk, ideally. (Measurement error occurs too of course, but that issue is outside this answer's scope somewhat, so aside from mentioning it here, I'll leave it alone otherwise.) Some risk always exists of making a false positive inference about the null being less useful than the alternative hypothesis, at least unless the inferrer knows the null is true. Only in the rather hard-to-conceive circumstance of knowledge that the null is literally true would an inference favoring an alternative hypothesis be definitely false...at least, as far as I can imagine at the moment.

Clearly, widespread usage or convention is not the best authority on epistemic or inferential validity. Even published resources are fallible; see for instance Fallacy in p-value definition. Your reference (Hurlbert & Lombardi, 2009) offers some interesting exposition of this principle too (page 322):

StatSoft (2007) boasts on their website that their online manual “is the only internet resource on statistics recommended by Encyclopedia Brittanica.” Never has it been so important to ‘Distrust Authority,’ as the bumper sticker says. [Comically broken URL converted to hyperlinked text.]

Another case in point: this phrase in a very recent Nature News article (Nuzzo, 2014): "P value, a common index for the strength of evidence..." See Wagenmakers' (2007, page 787) "Problem 3: $p$ Values Do Not Quantify Statistical Evidence"...However, @MichaelLew (Lew, 2013) disagrees in a way you might find useful: he uses $p$ values to index likelihood functions. Yet in as much as these published sources contradict one another, at least one must be wrong! (On some level, I think...) Of course, this is not as bad as "untrustworthy" per se. I hope I can coax Michael into chiming in here by tagging him as I have (but I'm not sure user tags send notifications when edited in – I don't think yours in the OP did). He may be the only one who can save Nuzzo – even Nature itself! Help us Obi-Wan! (And forgive me if my answer here demonstrates that I've still failed to comprehend the implications of your work, which I'm sure I have in any case...) BTW, Nuzzo also offers some intriguing self-defense and refutation of Wagenmaakers' "Problem 3": see Nuzzo's "Probable cause" figure and supporting citations (Goodman, 2001, 1992; Gorroochurn, Hodge, Heiman, Durner, & Greenberg, 2007). These just might contain the answer you're really looking for, but I doubt I could tell.

Re: your multiple choice question, I select d. You may have misinterpreted some concepts here, but you're certainly not alone if so, and I'll leave the judgment to you, as only you know what you really believe. Misinterpretation implies some amount of certainty, whereas asking a question implies the opposite, and that impulse to question when uncertain is quite laudable and far from ubiquitous, unfortunately. This matter of human nature makes the incorrectness of our conventions sadly short of harmless, and deserving of complaints such as those referenced here. (Thanks in part to you!) However, your proposal is not completely correct either.

Some interesting discussion of problems related to $p$ values in which I've participated appears in this question: Accommodating entrenched views of p-values. My answer lists a few references you may find useful for reading further into the interpretive problems and alternatives to $p$ values. Be forewarned: I still haven't hit the bottom of this particular rabbit hole myself, but I can at least tell you that it's very deep. I'm still learning about it myself (else I suspect I'd be writing from a more Bayesian perspective [edit]: or maybe the NFSA perspective! Hurlbert & Lombardi, 2009), I am a weak authority at best, and I welcome any corrections or elaborations others may offer to what I've said here. All I can opine in conclusion is that there probably is a mathematically correct answer, and it may well be that most people get it wrong. The right answer certainly doesn't come easily, as the following references demonstrate...

P.S. As requested (sort of...I admit I'm really just tacking this on instead of working it in), this question is a better reference for the sometimes uniform distribution of $p$ given the null: "Why are p-values uniformly distributed under the null hypothesis?" Of particular interest are @whuber's comments, which raise a class of exceptions. As is somewhat true with the discussion as a whole, I don't follow the arguments 100%, let alone their implications, so I'm not sure those problems with $p$ distribution uniformity are actually exceptional. Further cause for deep-seated statistical confusion, I'm afraid...

References

- Goodman, S. N. (1992). A comment on replication, P‐values and evidence. Statistics in Medicine, 11(7), 875–879.
- Goodman, S. N. (2001). Of P-values and Bayes: A modest proposal. Epidemiology, 12(3), 295–297. Retrieved from http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/ETP_Cetacean_Assessment/Of_P_Values_and_Bayes__A_Modest_Proposal.6.pdf.
- Goodman, S. (2008). A dirty dozen: Twelve P-value misconceptions. Seminars in Hematology, 45(3), 135–140. Retrieved from http://xa.yimg.com/kq/groups/18751725/636586767/name/twelve+P+value+misconceptions.pdf.
- Gorroochurn, P., Hodge, S. E., Heiman, G. A., Durner, M., & Greenberg, D. A. (2007). Non-replication of association studies: “pseudo-failures” to replicate? Genetics in Medicine, 9(6), 325–331. Retrieved from http://www.nature.com/gim/journal/v9/n6/full/gim200755a.html.
- Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman–Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. Retrieved from http://xa.yimg.com/kq/groups/1542294/508917937/name/HurlbertLombardi2009AZF.pdf.
- Lew, M. J. (2013). To P or not to P: On the evidential nature of P-values and their place in scientific inference. arXiv:1311.0081 [stat.ME]. Retrieved from http://arxiv.org/abs/1311.0081.
- Moyé, L. A. (2008). Bayesians in clinical trials: Asleep at the switch. Statistics in Medicine, 27(4), 469–482.
- Nuzzo, R. (2014, February 12). Scientific method: Statistical errors. Nature News, 506(7487). Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700.
- Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. Retrieved from http://www.brainlife.org/reprint/2007/Wagenmakers_EJ071000.pdf.

share|improve this answer
    
I'm still working through your very thorough answer (thanks for it), but your mention of the "Bayesian invasion" made me think of "Bayesians in Clinical Trials: Asleep at the Switch", reprinted as Chapter 12 here, which I'm also slowly wrapping my head around. –  Andrew Klaassen Feb 27 at 23:28
    
"You have either reached a page that is unavailable for viewing or reached your viewing limit for this book"...? –  Nick Stauner Feb 27 at 23:31
1  
That's unfortunate. If you have journal access, you can also find it here. A search for the phrase "bayesians are now breaking down traditional barriers in clinical trials" might also get you there. –  Andrew Klaassen Feb 27 at 23:33
1  
Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian also contains an entertaining history of p-values and attack on the uses of Bayesian analysis in research. I can't say I understand it well enough to evaluate it, but I figure it's good to at least be aware of correctives to current enthusiasms. –  Andrew Klaassen Feb 28 at 0:27
1  
@NickStauner Just found this discussion. It is not necessary for at least one account to be wrong if there is a set of accounts that disagree. They may be based on different models. [If you are game, you should read Bill Thompson's book The Nature of Statistical Evidence (2005).] Nonethless, my account is definitely right ;-) (Although just this morning rejected by a journal again.) I found the Nuzzo paper to be careless and potentially misleading. –  Michael Lew Apr 4 at 3:46
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.