Using p-value to compute the probability of hypothesis being true; what else is needed?

Question

Question:

One common misunderstanding of p-values is that they represent the probability of the null hypothesis being true. I know that's not correct and I know that p-values only represent the probability of finding a sample as extreme as this, given that the null hypothesis is true. However, intuitively, one should be able to derive the first from the latter. There must be a reason why no-one is doing this. What information are we missing that restricts us from deriving the probability of hypothesis being true from p-value and related data?

---

Example:

Our hypothesis is "Vitamin D affects mood" (null hypothesis being "no effect"). Let's say that we perform an appropriate statistical study with 1000 people and find a correlation between mood and vitamin levels. All other things being equal, a p-value of 0.01 indicates higher likelihood of true hypothesis than a p-value of 0.05. Let's say we get a p-value of 0.05. Why can't we calculate the actual probability that our hypothesis is true? What information are we missing?

---

Alternate terminology for frequentist statisticians:

If you accept the premise of my question, you can stop reading here. The following is for people who refuse to accept that a hypothesis can have a probability interpretation. Let's forget the terminology for a moment. Instead...

Let's say you are betting with your friend. Your friend shows you a thousand statistical studies about unrelated subjects. For each study you are only allowed to look at the p-value, sample size, and standard deviation of the sample. For each study, your friend offers you some odds to bet that the hypothesis presented in the study is true. You can choose to either take the bet or not take it. After you have made bets for all 1000 studies, an oracle ascends upon you and tells you which hypothesis are correct. This information allows you to settle the bets. My claim is that there exists an optimal strategy for this game. In my worldview that's equivalent to knowing probabilities for hypothesis being true, but if we disagree on that, it's fine. In that case we can simply talk about ways to employ p-values to maximize expectation for the bets.

Answer 1

Other answers get all philosophical, but I don't see why it is needed here. Let's consider your example:

Our hypothesis is "Vitamin D affects mood" (null hypothesis being "no effect"). Let's say that we perform an appropriate statistical study with 1000 people and find a correlation between mood and vitamin levels. All other things being equal, a p-value of 0.01 indicates higher likelihood of true hypothesis than a p-value of 0.05. Let's say we get a p-value of 0.05. Why can't we calculate the actual probability that our hypothesis is true? What information are we missing?

For $n=1000$, getting $p=0.05$ corresponds to the sample correlation coefficient $\hat \rho=0.062$. The null hypothesis is $H_0: \rho=0$. The alternative hypothesis is $H_1: \rho\ne 0$.

The p-value is $$p\text{-value} = P\big(|\hat\rho|\ge 0.062 \;\big|\; \rho=0\big),$$ and we can compute it based on the sampling distribution of $\hat\rho$ under the null; nothing else is needed.

You want to compute $$P(H_0\;|\;\text{data})=P\big(\rho=0\;\big|\; \hat\rho= 0.062\big),$$

and for this you need the whole bunch of additional ingredients. Indeed, by applying Bayes theorem we can rewrite it as follows:

$$\frac{P\big( \hat\rho= 0.062 \;\big|\;\rho=0\big) \cdot P(\rho=0)}{P\big( \hat\rho= 0.062 \;\big|\;\rho=0\big) \cdot P(\rho=0)+P\big( \hat\rho= 0.062 \;\big|\;\rho\ne0\big) \cdot (1-P(\rho=0))}.$$

So to compute the posterior probability of the null you need to have two additional things:

1. Prior that the null hypothesis is true: $P(\rho=0)$.
2. Assumption about how $\rho$ is distributed if the alternative hypothesis is true. This is needed to compute the $P\big( \hat\rho= 0.062 \;\big|\;\rho\ne0\big)$ term.

If you are willing to assume that $P(\rho=0)=0.5$ --- even though I personally am not sure why this should ever be a meaningful assumption, --- you will still need to assume the distribution of $\rho$ under alternative. In this case, you will be able to compute something called Bayes factor:

$$B=\frac{P\big( \hat\rho= 0.062 \;\big|\;\rho=0\big) }{P\big( \hat\rho= 0.062 \;\big|\;\rho\ne0\big)}.$$

As you see, Bayes factor does not depend on the prior probability of the null, but it does depend on the prior probability of $\rho$ (under the alternative).

[Please note that the nominator in the Bayes factor is not the p-value, because of the equality instead of the inequality sign. So when computing Bayes factor or $P(H_0)$ we are not using the p-value itself at all. But we are of course using the sampling distribution $P(\hat\rho\;|\;\rho=0)$.]

Answer 2

Quid est veritas?

I can accept @amoeba's answer as readily as the original poster. I caution, however, that in all my work, I've not encountered a Bayesian analysis which calculated "the probability that the null hypothesis is true". And such a conclusion would attract a whole host of arguments from those reviewing your work! Philosophically, it does bring us back to the question: "what is truth?" Perhaps "truth" is irrefutable, even to evidence itself. Statistics is a tool of science to quantify uncertainty. I still maintain that, while evidence can strongly point to a truth, there is always a risk of a false positive finding, and the Good Statistician should report this risk. Even in Bayesian decision theoretic testing, a decision rule is given so that we can accept or reject hypotheses based on Bayes factors which are roughly proportional to $Pr(H_0 | X)$, but our belief is never $1$ or $0$ even when our decision is. Decision theory gives us a means of "going forward" with partial knowledge and accepting these risks.

Part of the rationale for null hypothesis statistical testing (NHST) and the $p$-value is Karl Popper's philosophy of falsification. In this: a critical assumption is that the "truth" is never known, we can only whittle down other hypotheses. An interesting and a valid criticism of NHST is that you are forced to make ridiculous assumptions, like that smoking does not cause cancer when you're really interested in a descriptive (not inferential) study: and you are merely describing how much cancer smoking causes.

The converse criticism has been applied to Bayesian studies where you can liberally apply priors: Dennis Lindley has said, "With prior probability 0 that the moon is made of cheese, astronauts returning with arms full of cheese still could not convince."

The missing information to determine whether the null hypothesis is true is, trivially, the knowledge as to whether the null hypothesis is true. Ironically, when focused on descriptive statistics, we can accept tolerable ranges of possible effects and conclude somewhat strongly that a trend is probably true: but statistical testing does not lead us to such findings. Even in Bayesian inference, no data will lead to a singular posterior without having some methodologic issues, so incorporation of a prior does not fix this problem.

Answer 3

There are two attempts to do exactly what you have said in statistical history, the Bayesian and the Fiducial. R. A. Fisher founded two schools of statistical thinking, the Likelihoodist school built around the method of maximum likelihood and the Fiducial, which ended in failure but which attempts to do exactly what you want.

The short answer as to why it failed is that its probability distributions did not end up integrating to unity. The lesson, in the end, was that the prior probability is a necessary thing to have to create what you are trying to create. Indeed, you are going right down the path of one of history’s greatest statisticians and more than a few of the other greats died hoping for a resolution to this problem. If it were found it would place null hypothesis methods on par with Bayesian methods in terms of the types of problems that they could solve. Indeed, it would push past Bayes except where real prior information existed.

You also want to be careful with your statement that a p-value indicates a higher likelihood for the alternative. That is only true in the Fisherian Likelihoodist school. It is not at all true in the Pearson-Neyman Frequentist school. Your bet at the bottom appears to be a Pearson-Neyman bet while your p-value is incompatible as it is coming from the Fisherian school.

To be charitable I am going to assume, that for your example, that there is no publication bias and so only significant results appear in journals creating a high false discovery rate. I am treating this as a random sample of all studies performed, regardless of the results. I would argue that your betting odds would not be coherent in the classical de Finetti sense of the word.

In de Finetti’s world, a bet is coherent if the bookie cannot be gamed by players so that they face a sure loss. In the simplest construction, it is like the solution to the problem of cutting the cake. One person cuts the piece in half, but the other person chooses which piece they want. In this construction one person would state the prices for the bets on each hypothesis, but the other person would choose to either buy or sell the bet. In essence, you could short sell the null. To be optimal, the odds would have to be strictly fair. P-values do to not lead to fair odds.

To illustrate this, consider the study by Wetzels, et al at http://ejwagenmakers.com/2011/WetzelsEtAl2011_855.pdf

The citation for which is: Ruud Wetzels, Dora Matzke, Michael D. Lee, Jeffrey N. Rounder, Geoffrey J. Iverson and Eric-Jan Wagenmakers. Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests. Perspectives on Psychological Science. 6(3) 291-298. 2011

This is a direct comparison of 855 published t-tests using Bayes factors to bypass the problem of the prior distribution. In 70% of the p-values between .05 and .01, the Bayes factors were at best, anecdotal. This is due to the mathematical form used by Frequentists to solve the problem.

Null hypothesis methods presume that the model is true and by their construction use a minimax statistical distribution rather than a probability distribution. Both of these factors impact differences between Bayesian and non-Bayesian solutions. Consider a study where the Bayesian method evaluates the posterior probability of a hypothesis as three percent. Imagine that the p-value is less than five percent. Both are true since three percent is less than five percent. Nonetheless, the p-value isn’t a probability. It only states the maximum value that could be the probability of seeing the data, not the actual probability a hypothesis is true or false. Indeed, under the p-value construction, you cannot distinguish between effects due to chance with a true null and a false null with good data.

If you look at the Wetzel study, you will note that it is very obvious that the odds implied by the p-values do not match the odds implied by the Bayesian measure. Since the Bayesian measure is both admissible and coherent, and the non-Bayesian is not coherent, it is not safe to assume the p-values map to the true probabilities. The forced assumption that the null is valid provides nice coverage probabilities, but it does not produce nice gambling probabilities.

To get a better feel as to why, consider Cox’s first axiom that the plausibility of a hypothesis can be described by a real number. Implicitly, this means that all hypothesis have a real number tied to their plausibility. In null hypothesis methods, only the null has a real number tied to its plausibility. The alternative hypothesis has no measurement made and it is certainly not the complement to the probability of observing the data given that the null is true. Indeed, if the null is true, then the complement is false by assumption without regard to the data.

If you constructed the probabilities using p-values as the basis of your measurement, then the Bayesian using Bayesian measurements would always be capable of getting an advantage over you. If the Bayesian set the odds then Pearson and Neyman decision theory would provide a statement of bet or do not bet, but they would not be able to define the amount to bet. As the Bayesian odds were fair, the expected gain from using Pearson and Neyman’s method would be zero.

Indeed, the Wetzel study is really what you are talking about doing, but with 145 fewer bets. If you look at table three you will see some studies where the Frequentist rejects the null, but the Bayesian finds that the probability favors the null.

Answer 4

A frequentist analysis cannot give you the probability that a particular hypothesis is true (or false) because it has no long run frequency (it is either true or it isn't) so we cannot assign a probability to it (except perhaps 0 or 1). If you want to know the probability that a particular hypothesis is true, we need to adopt a Bayesian framework (where it is straightforward, we just need to consider the prior probabilities etc.).

Frequentists can find optimal strategies for acting on null hypothesis tests (Neyman-Pearson framework) but they can't translate that into a probability that the hypothesis is true, but only because of their definition of a probability.

Answer 5

After you have made bets for all 1000 studies, an oracle ascends upon you and tells you which hypothesis are correct. This information allows you to settle the bets. My claim is that there exists an optimal strategy for this game.

The problem in your setup is the Oracle. It doesn't usually come to settle the bets. Say, you are betting that the probability that it is true that smoking causes cancer is 97%. When will this Oracle come to settle the bet? Never. Then how would you prove that your optimal strategy optimal?

However, if you remove an Oracle, and introduce other agents such as competitors and customers, then there would be an optimal strategy. I'm afraid it won't be based on p-values, though. It would be more similar to Gosset's approach with loss functions. For instance, you and your competitors in farming sector are betting on the weather forecast being true. Whoever picks a better strategy is going to make more money. There's no need in Oracle, and the bets are settled on the markets. You can't base strategy on p-values here, you have to account for losses and profits in dollars.

Answer 6

In hypothesis you want to test some statement about the real world, e.g. the average length of all men is 1.75m. We would then formulate a hypothesis test like e.g. $H_0: \mu_L=1.75$ versus $H_1: \mu_L \ne 1.75$.

This is our statement and we want to test whether in the real world this is a fact. But frequentists state that in the real world this is either true or false. As in the real world $H_0$ is either true or false, this means that in the real world $P(H_0=TRUE)$ is either 0 or 1.

So in theory the result of our hypothesis test should be $H_0$ is true or false but as we only work on a sample we can not make such hard conclusions, therefore we try to use some statistical variant of a mathematical technique called 'proof by contradiction'. For detail see What follows if we fail to reject the null hypothesis?.

For a thread on p-values see Misunderstanding a P-value?

Baysians do something different; they express their belief or credibility they have in their conclusion of the test, so it is not realy the probability that $H_0$ is true, but more the degree of belief they have in their conclusion they make after the test about $H_0$. This is why it is called ''credibility''.

Taking your example, you test "$H_0:$ Vitamin D affects mood" versus "$H_1:$ Vitamin D doe not affect mood".

Based on a sample you compute some test-statistic and its probability of being exceeded when $H_0$ is true. If this value of the test statistic is very low (below our chosen significance level) then assuming that $H_0$ is true leads to something very improbable or it leads so to say to ''a statistical contradiction'' and

Frequentists will conclude that in such case $H_0$ leads to statistical non-sense. However, in the ''real world'' there is only one truth $H_0$ or $H_1$ !

Bayesians compute the probability that $H_0$ is true given the data. So there also, in the real world, $H_0$ is true or $H_1$ is true, but using data they can express their degree of belief (derived from the data) that $H_0$ is true.

They call this the ''credibility of the hypothesis'', but it does not say anything about the probability that $H_0$ is true (nor about the probability that $H_1$ is true)

They just express their belief in their ''conclusion of the test'' derived from ''available data''.