Why are 0.05 < p < 0.95 results called false positives?

Question

Edit: The basis of my question is flawed, and I need to spend some time figuring out whether it can even be made to make sense.

Edit 2: Clarifying that I recognize that a p-value isn't a direct measure of the probability of a null hypothesis, but that I assume that the closer a p-value is to 1, the more likely it is that a hypothesis has been chosen for experimental testing whose corresponding null hypothesis is true, while the closer a p-value is to 0, the more likely it is that a hypothesis has been chosen for experimental testing whose corresponding null hypothesis is false. I can't see how this is false unless the set of all hypotheses (or all hypotheses picked for experiments) is somehow pathological.

Edit 3: I think I'm still not using clear terminology to ask my question. As lottery numbers are read out, and you match them to your ticket one-by-one, something changes. The probability that you have won does not change, but the probability that you can turn the radio off does. There's a similar change that happens when experiments are done, but I have a feeling that the terminology I'm using - "p-values change the likelihood that a true hypothesis has been chosen" - isn't the correct terminology.

Edit 4: I've received two amazingly detailed and informative answers that contain a wealth of information for me to work through. I'll vote them both up now and then come back to accept one when I've learned enough from both answers to know that they've either answered or invalidated my question. This question opened a much bigger can of worms than the one I was expecting to eat.

In papers I've read, I've seen results with p > 0.05 after validation called "false positives". However, isn't it still more likely than not that I've chosen a hypothesis to test with a false corresponding null hypothesis when the experimental data has a p < 0.50 which is low but > 0.05, and aren't both the null hypothesis and the research hypothesis statistically uncertain/insignificant (given the conventional statistical significance cutoff) anywhere between 0.05 < p < 0.95 whatever the inverse of p < 0.05 is, given the asymmetry pointed out in @NickStauner's link?

Let's call that number A, and define it as the p-value which says the same thing about the likelihood that you've picked a true null hypothesis for your experiment/analysis that a p-value of 0.05 says about the likelihood that you've picked a true non-null hypothesis for your experiment/analysis. Doesn't 0.05 < p < A just say, "Your sample size wasn't big enough to answer the question, and you won't be able to judge application/real-world significance until you get a bigger sample and get your statistical significance sorted out"?

In other words, shouldn't it be correct to call a result definitely false (rather than simply unsupported) if and only if p > A?

This seems straightforward to me, but such widespread usage tells me that I might be wrong. Am I:

a) misinterpreting the mathematics,
b) complaining about a harmless-if-not-exactly-correct convention,
c) completely correct, or
d) other?

I recognize that this sounds like a call for opinions, but this seems like a question with a definite mathematically correct answer (once a significance cutoff is set) that either I or (almost) everybody else is getting wrong.

Answer 1

Your question is based on a false premise:

isn't the null hypothesis still more likely than not to be wrong when p < 0.50

A p-value is not a probability that the null hypothesis is true. For example, if you took a thousand cases where the null hypothesis is true, half of them will have p < .5. Those half will all be null.

Indeed, the idea that p > .95 means that the null hypothesis is "probably true" is equally misleading. If the null hypothesis is true, the probability that p > .95 is exactly the same as the probability that p < .05.

ETA: Your edit makes it clearer what the issue is: you still do have the issue above (that you're treating a p-value as a posterior probability, when it is not). It's important to note that this is not a subtle philosophical distinction (as I think you're implying with your discussion of the lottery tickets): it has enormous practical implications for any interpretation of p-values.

But there is a transformation you can perform on p-values that will get you to what you're looking for, and it's called the local false discovery rate. (As described by this nice paper, it's the frequentist equivalent of the "posterior error probability", so think of it that way if you like).

Let's work with a concrete example. Let's say you are performing a t-test to determine whether a sample of 10 numbers (from a normal distribution) has a mean of 0 (a one-sample, two-sided t-test). First, let's see what the p-value distribution looks like when the mean actually is zero, with a short R simulation:

null.pvals = replicate(10000, t.test(rnorm(10, mean=0, sd=1))$p.value)
hist(null.pvals)

As we can see, null p-values have a uniform distribution (equally likely at all points between 0 and 1). This is a necessary condition of p-values: indeed, it's precisely what p-values mean! (Given the null is true, there is a 5% chance it is less than .05, a 10% chance it is less than .1...)

Now let's consider the alternative hypothesis- cases where the null is false. Now, this is a bit more complicated: when the null is false, "how false" is it? The mean of the sample isn't 0, but is it .5? 1? 10? Does it randomly vary, sometimes small and sometimes large? For simplicity's sake, let's say it is always equal to .5 (but remember that complication, it'll be important later):

alt.pvals = replicate(10000, t.test(rnorm(10, mean=.5, sd=1))$p.value)
hist(alt.pvals)

Notice that the distribution is now not uniform: it is shifted towards 0! In your comment you mention an "asymmetry" that gives information: this is that asymmetry.

So imagine you knew both of those distributions, but you're working with a new experiment, and you also have a prior that there's a 50% chance it's null and 50% that it's alternative. You get a p-value of .7. How can you get from that and the p-value to a probability?

What you should do is compare densities:

lines(density(alt.pvals, bw=.02))
plot(density(null.pvals, bw=.02))

And look at your p-value:

abline(v=.7, col="red", lty=2)

That ratio between the null density and the alternative density can be used to calculate the local false discovery rate: the higher the null is relative to the alternative, the higher the local FDR. That's the probability that the hypothesis is null (technically it has a stricter frequentist interpretation, but we'll keep it simple here). If that value is very high, then you can make the interpretation "the null hypothesis is almost certainly true." Indeed, you can make a .05 and .95 threshold of the local FDR: this would have the properties you're looking for. (And since local FDR increases monotonically with p-value, at least if you're doing it right, these will translate to some thresholds A and B where you can say "between A and B we are unsure").

Now, I can already hear you asking "then why don't we use that instead of p-values?" Two reasons:

1. You need to decide on a prior probability that the test is null
2. You need to know the density under the alternative. This is very difficult to guess at, because you need to determine how large your effect sizes and variances can be, and how often they are so!

You do not need either of those for a p-value test, and a p-value test still lets you avoid false positives (which is its primary purpose). Now, it is possible to estimate both of those values in multiple hypothesis tests, when you have thousands of p-values (such as one test for each of thousands of genes: see this paper or this paperfor instance), but not when you're doing a single test.

Finally, you might say "Isn't the paper still wrong to say a replication that leads to a p-value above .05 is necessarily a false positive?" Well, while it's true that getting one p-value of .04 and another p-value of .06 doesn't really mean the original result was wrong, in practice it's a reasonable metric to pick. But in any case, you might be glad to know others have their doubts about it! The paper you refer to is somewhat controversial in statistics: this paper uses a different method and comes to a very different conclusion about the p-values from medical research, and then that study was criticized by some prominent Bayesians (and round and round it goes...). So while your question is based on some faulty presumptions about p-values, I think it does examine an interesting assumption on the part of the paper you cite.

Answer 2

^{Hover your mouse over any tag ($\leftarrow$ is a fake tag) appearing below to see a brief excerpt of its wiki. Please forgive the disruption of line spacing. I find it worthwhile because tag excerpts may help readers to check understanding of jargon while reading through. Some of these excerpts may deserve editing as well, so they also deserve a publicist, IMHO.}

$p>.05$ ordinarily implies one should not reject the null-hypothesis. Conversely, type-i-errors or false positives occur when one does reject the null due to sampling error or some other unusual incident that produces a sample that was otherwise unlikely (usually with $p<.05$) to have been sampled randomly from a population in which the null is true. A result with $p>.05$ that is called a false positive seems to reflect a misunderstanding of null hypothesis significance-testing (NHST). Misunderstandings are not uncommon in published research literature, as NHST is notoriously counter-intuitive. This is one of the rallying cries of the bayesian invasion (which I support, but do not follow...yet). I have worked with mistaken impressions such as these myself until recently, so I sympathize most heartily.

@DavidRobinson is correct in observing that $p$ is not the probability of the null being false in frequentist NHST. This is (at least) one of Goodman's ⁽²⁰⁰⁸⁾ "Dirty Dozen" misconceptions about $p$ values ^{(see also Hurlbert & Lombardi, 2009)}. In NHST, $p$ is the probability that one would draw any future random samples by the same means that would exhibit a relationship or difference (or whatever effect-size is being tested against the null, if other varieties of effect size exist...?) at least as different from the null hypothesis as the sample(s) from the same population(s) one has tested to arrive at a given $p$ value, if the null is true. That is, $p$ is the probability of obtaining a sample like yours given the null; it does not reflect the probability of the null – at least, not directly. Conversely, Bayesian methods pride themselves on their formulation of statistical analyses as focused on estimating the evidence for or against a prior theory of an effect given the data, which they argue is a more intuitively appealing approach ^{(Wagenmakers, 2007)}, among other advantages, and setting aside debatable disadvantages. (To be fair, see "What are the cons of Bayesian analysis?" You have also commented to cite articles that might offer some nice answers there: ^{Moyé, 2008; Hurlbert & Lombardi, 2009.)}

Arguably, the null hypothesis as literally stated is often more likely than not to be wrong, because null hypotheses are most commonly, literally hypotheses of zero effect. (For some handy counter-examples, see answers to: "Are large data sets inappropriate for hypothesis testing?") Philosophical issues such as the butterfly effect threaten the literal validity of any such hypothesis; hence the null is useful most generally as a basis of comparison for an alternative hypothesis of some nonzero effect. Such an alternative hypothesis may remain more plausible than the null after data have been collected that would've been improbable if the null were true. Hence researchers typically infer support for an alternative hypothesis from evidence against the null, but that is not what p-values quantify directly ^{(Wagenmakers, 2007)}.

As you suspect, statistical-significance is a function of sample-size, as well as effect size and consistency. (See @gung's answer to the recent question, "How can a t-test be statistically significant if the mean difference is almost 0?") The questions we often intend to ask of our data are, "What is the effect of x on y?" For various reasons (including, IMO, misconceived and otherwise deficient educational programs in statistics, especially as taught by non-statisticians), we often find ourselves instead asking literally the loosely related question, "What is the probability of sampling data such as mine randomly from a population in which x does not affect y?" This is the essential difference between effect size estimation and significance testing, respectively. A $p$ value answers only the latter question directly, but several professionals (@rpierce could probably give you a better list than I; forgive me for dragging you into this!) have argued that researchers misread $p$ as an answer to the former question of effect size all too often; I'm afraid I must agree.

To respond more directly regarding the meaning of $.05<p<.95$, it is that the probability of sampling data randomly from a population of which the null is true, but that exhibits a relationship or difference that differs from that which the null describes literally by at least as wide and consistent a margin as your data does...< inhale>...is between 5–95%. One may certainly argue this is a consequence of sample size, because increasing sample size improves one's ability to detect small and inconsistent effect sizes and differentiate them from a null of, say, zero effect with confidence exceeding 5%. However, small and inconsistent effect sizes may or may not be significant pragmatically ( $\ne$ significant statistically – another of Goodman's (2008) dirty dozen); this depends far more on the meaning of the data, with which statistical significance only concerns itself to a limited extent. See my answer to the above.

Shouldn't it be correct to call a result definitely false (rather than simply unsupported) if...p > 0.95?

Since data should usually represent empirically factual observations, they should not be false; only inferences about them should face this risk, ideally. (Measurement error occurs too of course, but that issue is outside this answer's scope somewhat, so aside from mentioning it here, I'll leave it alone otherwise.) Some risk always exists of making a false positive inference about the null being less useful than the alternative hypothesis, at least unless the inferrer knows the null is true. Only in the rather hard-to-conceive circumstance of knowledge that the null is literally true would an inference favoring an alternative hypothesis be definitely false...at least, as far as I can imagine at the moment.

Clearly, widespread usage or convention is not the best authority on epistemic or inferential validity. Even published resources are fallible; see for instance Fallacy in p-value definition. Your reference ^{(Hurlbert & Lombardi, 2009)} offers some interesting exposition of this principle too ^{(page 322):}

StatSoft (2007) boasts on their website that their online manual “is the only internet resource on statistics recommended by Encyclopedia Brittanica.” Never has it been so important to ‘Distrust Authority,’ as the bumper sticker says. [Comically broken URL converted to hyperlinked text.]

Another case in point: this phrase in a very recent Nature News article ^{(Nuzzo, 2014)}: "P value, a common index for the strength of evidence..." See Wagenmakers' ^{(2007, page 787)} "Problem 3: $p$ Values Do Not Quantify Statistical Evidence"...However, @MichaelLew ^{(Lew, 2013)} disagrees in a way you might find useful: he uses $p$ values to index likelihood functions. Yet in as much as these published sources contradict one another, at least one must be wrong! (On some level, I think...) Of course, this is not as bad as "untrustworthy" per se. _{I hope I can coax Michael into chiming in here by tagging him as I have (but I'm not sure user tags send notifications when edited in – I don't think yours in the OP did). He may be the only one who can save Nuzzo – even Nature itself! Help us Obi-Wan! (And forgive me if my answer here demonstrates that I've still failed to comprehend the implications of your work, which I'm sure I have in any case...)} BTW, Nuzzo also offers some intriguing self-defense and refutation of Wagenmaakers' "Problem 3": see Nuzzo's "Probable cause" figure and supporting citations ^{(Goodman, 2001, 1992; Gorroochurn, Hodge, Heiman, Durner, & Greenberg, 2007)}. These just might contain the answer you're really looking for, but I doubt I could tell.

Re: your multiple choice question, I select d. You may have misinterpreted some concepts here, but you're certainly not alone if so, and I'll leave the judgment to you, as only you know what you really believe. Misinterpretation implies some amount of certainty, whereas asking a question implies the opposite, and that impulse to question when uncertain is quite laudable and far from ubiquitous, unfortunately. This matter of human nature makes the incorrectness of our conventions sadly short of harmless, and deserving of complaints such as those referenced here. (Thanks in part to you!) However, your proposal is not completely correct either.

Some interesting discussion of problems related to $p$ values in which I've participated appears in this question: Accommodating entrenched views of p-values. My answer lists a few references you may find useful for reading further into the interpretive problems and alternatives to $p$ values. Be forewarned: I still haven't hit the bottom of this particular rabbit hole myself, but I can at least tell you that it's very deep. I'm still learning about it myself (else I suspect I'd be writing from a more Bayesian perspective [edit]: or maybe the NFSA perspective! ^{Hurlbert & Lombardi, 2009)}, I am a weak authority at best, and I welcome any corrections or elaborations others may offer to what I've said here. All I can opine in conclusion is that there probably is a mathematically correct answer, and it may well be that most people get it wrong. The right answer certainly doesn't come easily, as the following references demonstrate...

P.S. As requested (sort of...I admit I'm really just tacking this on instead of working it in), this question is a better reference for the sometimes uniform distribution of $p$ given the null: "Why are p-values uniformly distributed under the null hypothesis?" Of particular interest are @whuber's comments, which raise a class of exceptions. As is somewhat true with the discussion as a whole, I don't follow the arguments 100%, let alone their implications, so I'm not sure those problems with $p$ distribution uniformity are actually exceptional. Further cause for deep-seated statistical confusion, I'm afraid...

References

<sub>- Goodman, S. N. (1992). A comment on replication, P‐values and evidence. Statistics in Medicine, 11(7), 875–879.
- Goodman, S. N. (2001). Of P-values and Bayes: A modest proposal. Epidemiology, 12(3), 295–297. Retrieved from http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/ETP_Cetacean_Assessment/Of_P_Values_and_Bayes__A_Modest_Proposal.6.pdf.
- Goodman, S. (2008). A dirty dozen: Twelve P-value misconceptions. Seminars in Hematology, 45(3), 135–140. Retrieved from http://xa.yimg.com/kq/groups/18751725/636586767/name/twelve+P+value+misconceptions.pdf.
- Gorroochurn, P., Hodge, S. E., Heiman, G. A., Durner, M., & Greenberg, D. A. (2007). Non-replication of association studies: “pseudo-failures” to replicate? Genetics in Medicine, 9(6), 325–331. Retrieved from http://www.nature.com/gim/journal/v9/n6/full/gim200755a.html.
- Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman–Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. Retrieved from http://xa.yimg.com/kq/groups/1542294/508917937/name/HurlbertLombardi2009AZF.pdf.
- Lew, M. J. (2013). To P or not to P: On the evidential nature of P-values and their place in scientific inference. arXiv:1311.0081 [stat.ME]. Retrieved from http://arxiv.org/abs/1311.0081.
- Moyé, L. A. (2008). Bayesians in clinical trials: Asleep at the switch. Statistics in Medicine, 27(4), 469–482.
- Nuzzo, R. (2014, February 12). Scientific method: Statistical errors. Nature News, 506(7487). Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700.
- Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. Retrieved from http://www.brainlife.org/reprint/2007/Wagenmakers_EJ071000.pdf.</sub>