Timeline for Frequentist vs bayesian and P(data | H0) vs P(H0 | data) giving same result
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 21, 2021 at 15:32 | comment | added | Frank Harrell | I'm implying that it is not relevant to know this in such a highly specialized case. But you said it well. We need to also keep the "fixed sample size one look" in every comparison. But the bottom line is that if you believe that probabilities do not need to be relative frequencies, be Bayesian and don't try to jump back and forth. If you believe that the only basis for probabilities is frequencies, then don't attempt any comparison because Bayes has to be meaningless to you. | |
| Jul 21, 2021 at 12:00 | comment | added | Tim | @FrankHarrell it merely answers "why" the observed results ended up close. It was already explained in the comments that they in essence are different, though I edited the answer to make it explicit. | |
| Jul 21, 2021 at 11:58 | history | edited | Tim | CC BY-SA 4.0 |
added 494 characters in body
|
| Jul 21, 2021 at 11:15 | comment | added | Frank Harrell | I don't see the utility in this exercise. As others have eloquently said frequentism does not even conceive of probabilities that are not limits of long-term frequencies of event occurrences. And the equivalence only holds in a very special case where the sample size is fixed and there is exactly one look at the data. | |
| Jul 21, 2021 at 10:37 | history | edited | Tim | CC BY-SA 4.0 |
added 293 characters in body
|
| Jul 21, 2021 at 9:55 | comment | added | Tim | @jcp I added an example on the bottom of the answer for clarity. As for "the math" it sounds like something for a separate question. If your math tells you that the distribution of independent random variables is the same as if they were dependent, then by definition something is wrong. | |
| Jul 21, 2021 at 9:51 | history | edited | Tim | CC BY-SA 4.0 |
added 1717 characters in body
|
| Jul 21, 2021 at 9:48 | comment | added | jcp | Fair enough, but still it blows my mind that this is true under large amounts of data. Working out the math, I get to the point where if the priors are the same, $P(H0)$ and $P(H1)$ cancel out and I'm left with $P(H0|data) = p(data | H0)/(p(data | H0) + p(data | H1))$, which 1-only works out if $p(data | H0) + p(data | H1) = 1$ and 2-is independent on the prior choices | |
| Jul 21, 2021 at 9:30 | history | edited | Tim | CC BY-SA 4.0 |
added 368 characters in body
|
| Jul 21, 2021 at 9:22 | comment | added | Tim |
@jcp that's an overstatement. Keep in mind that no prior is truly "uninformative" and that in the $z$-test uses the normal approximation, not the exact distribution. For a counterexample, just change N=5 or a smaller value. Additionally, plot the beta distribution pdf's vs the normal approximation's and you'll see that they do differ. It's just a simple case and relatively large data for relatively weak prior, so both prior has small impace and the normal approximation works well enough.
|
|
| Jul 21, 2021 at 9:16 | comment | added | jcp | That's what I would expect. But: in my example, all possibilities of the test using a beta-binomial model with a "Haldane prior" leads $P(H0 | data) \tilde= P(data | H0)$. That leads me to believe that any classic p-value from a hypothesis test with Bernoulli random variables can be interpreted as approximately $P(H0 | data)$ with an Haldane prior | |
| Jul 21, 2021 at 8:42 | comment | added | Tim | @jcp no, see stats.stackexchange.com/questions/341553/… or stats.stackexchange.com/questions/275527/… Frequentist and Bayesian methods can give same or vary similar results in some cases but are not the same and their interpretation is different. | |
| Jul 21, 2021 at 8:10 | comment | added | jcp | But does that mean then that a p-value is almost equivalent to the posterior an uninformative prior? I can then interpret a p-value is virtually the probability that the H0 is true? | |
| Jul 21, 2021 at 8:04 | vote | accept | jcp | ||
| Jul 21, 2021 at 7:46 | comment | added | Tim | @jcp Edited second paragraph for this. You are not using a flat prior, but other "uninformative" prior, but let's say you used a flat prior $p(\theta) \propto 1$ then $p(\theta|X) \propto p(X|\theta) \times 1 = p(X|\theta)$. With other "uninformative" priors it's not exactly the same, but very close. Simply: with an "uninformative" prior posterior is nearly the same as likelihood. | |
| Jul 21, 2021 at 7:43 | history | edited | Tim | CC BY-SA 4.0 |
added 86 characters in body
|
| Jul 21, 2021 at 7:43 | comment | added | jcp | I understand that I'm comparing expected values but, as far as I understood, in the z-test I'm getting the prob that H0 is true given the differences of means (and std), while in the bayesian approach I estimate the distribution of the data and then compute the probability that H0 is true. My confusion comes from seeming to get P(H0|data) = P(data|H0) | |
| Jul 21, 2021 at 7:32 | history | answered | Tim | CC BY-SA 4.0 |