As the title suggests, Why is p-value termed as P(Data | Hypothesis/Model) and not P(Hypothesis | Data)? Shouldn't both be the same? Why is P(Data | Hypothesis) != P(Hypothesis | Data)? Is there any logical reasoning here that I am missing?
$\begingroup$
$\endgroup$
7
-
4$\begingroup$ $P(A\vert B)$ and $P(B\vert A)$ are often different and are related by Bayes’ Theorem: $P(B\vert A)=\dfrac{P(A\vert B)P(B)}{P(A)}$. $\endgroup$– DaveCommented Apr 9, 2020 at 22:14
-
$\begingroup$ @Dave sorry for any miss confusion caused, my question was why cant we term p-value as P(Hypothesis | Data)? because thats what we are doing we have a hypothesis and we want to find the probability of Data based on our hypothesis. $\endgroup$– christopherCommented Apr 9, 2020 at 22:21
-
2$\begingroup$ @DanishNazir p value would certainly be much more useful if it was equal to P(hypothesis | data). But we can't calculate that value without knowing the prior probability of both the data and the hypothesis. P(data | hypothesis) isn't nearly as useful, but it can be calculated directly from data, so a lot of people use it for lack of better options. (And far too many implicitly treat it as equivalent to P(hypothesis | data)). $\endgroup$– RayCommented Apr 10, 2020 at 19:50
-
$\begingroup$ Treating P(data | hypothesis)=P(hypothesis | data) wouldnt be wrong? i mean in what cases you would treat them equal $\endgroup$– christopherCommented Apr 10, 2020 at 19:52
-
2$\begingroup$ @DanishNazir That is nearly always wrong. Dave mentions Bayes' Theorem above. From that, it follows that P(data | hypothesis) = P(hypothesis | data) if and only if P(data) = P(hypothesis) (and both are nonzero), which generally is not the case. $\endgroup$– RayCommented Apr 10, 2020 at 19:54
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
1
The equivalence that you propose represents a fundamental (and often made) error, which the American Statistical Association has been trying to stamp out for some time now. See the statement by Wasserstein, Schirm & Lazar (2019).
You have one data set, but there are multiple competing hypotheses. The probability one might assign to one given hypothesis ought to reflect the relative strength of evidence for each hypothesis relative to the competitors. A p-value is nothing like that, in itself.
When you look at a hypothesis test generating a low p-value, it can seem plausible that the p-value represents the probability of the hypothesis. But think more generally. Imagine a hypothesis test involving a simple hypothesis with one parameter. That parameter is continuous in value, so the value lies on a continuum. There is a p-value that goes with every single one of the infinity of different values on that continuum, and some of those p-values would be quite high, very close to 1.0. The continuum collectively represents all possibilities for the value of the parameter. When we sum across all probabilities for an event, the sum should be 1. But summing all of these p-values will produce a number much, much larger than 1. Therefore, the p-values are not the probabilities that each different value of the parameter is correct or true.
-
$\begingroup$ This ingenious answer, far better than most I've seen to such questions, refers not to "the" p-value but to an infinite set of them -- one for each possible value of the parameter. So each p-value tells the strength of evidence against one of those null hypotheses. $\endgroup$– rolando2Commented Apr 10, 2020 at 18:46
$\begingroup$
$\endgroup$
6
The literal definition of a p-value is the probability of finding a test statistic at least as extreme as the observed test statistic, given that the null is true: something like (but not quite the same as) $P(\text{Data}\vert\text{Null is true})$. If such a result is unlikely (low p-value), we see that as evidence against the null hypothesis. Sufficiently strong evidence against the null hypothesis make us say, “No, we’re unlikely to get this result if the null is true, but we did get this result, so the null sure seems false,” kind of a proof by contradiction.
You claim that a p-value measures the probability of your null hypothesis given the data you’ve collected. This is a common mistake to make, but it is indeed a mistake.
-
$\begingroup$ Good explanation, one point though you are saying that with low p-value we can see that as evidence against the null hypothesis but if my p-value is e.g 0.0001 will that mean that my results are statistically significant? or in simple words will i get the same results 99.99% of the time? $\endgroup$ Commented Apr 9, 2020 at 22:31
-
2$\begingroup$ No, a small p-value like that tells you that you have a test statistic that you would not expect to see very often if the null hypothesis is true (such as a blizzard outside if it is summer). $\endgroup$– DaveCommented Apr 9, 2020 at 22:39
-
$\begingroup$ what do you mean by "test statistic" here? $\endgroup$ Commented Apr 9, 2020 at 22:41
-
1$\begingroup$ I mean the t-stat or F-stat or whatever you use in your particular hypothesis test. $\endgroup$– DaveCommented Apr 9, 2020 at 22:44
-
5$\begingroup$ The z-test vs the t-test isn’t an issue of whether or not you have data from a normal distribution. It is an issue of whether or not you know the variance of your distribution. I struggle to think of any scenario where we would know that. But yes, the z-stat is the test statistic for a z-test. $\endgroup$– DaveCommented Apr 9, 2020 at 22:53