5
$\begingroup$

I am drafting a teaching session for fellow clinicians to try and provide a somewhat intuitive understanding on how Bayesian statistics differs from the frequentist methods we are taught at medical school. In the session I propose an implausible intervention that is studied and happens to find a difference in sample means with a p-value of 0.048. We then go through comparing the 'oddness' of the results with the 'oddness' of the implausible intervention working. I describe Bayes method of comparing these oddities to try and find which is the least implausible. I turn to this calculator because it uses more concrete inputs than other formulations that need more abstract inputs such as p(data). I then go on to fiddle around with some of the inputs to help build an intuitive sense of what very low p-values achieve.

In my example I use it as follows:

  • Calculate the probability of: H0
  • Based on the probability of: p-value
  • P(H0): 0.999 (I use an average of the class's priors)
  • P(p-value|H0): 0.048 (this is the outcome p-value)
  • P(p-value|¬H0): 0.8 (I have assumed this is the same as study power)

Questions:

  1. Is P(p-value|¬H0) the same as a study power? Or is it only it only the same as the study power if the p-value = the "cut-off p-value". I.e, it varies with the observed p-value?
  2. If not, what is it and how could I estimate it and explain it?
  3. Is my methodology valid?
$\endgroup$
10
  • 2
    $\begingroup$ "Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence", James O. Berger and Thomas Sellke, Journal of the American Statistical Association, Vol. 82, No. 397 might be relevant here $\endgroup$ Commented Dec 29, 2023 at 14:18
  • 2
    $\begingroup$ The notation is confusing/incorrect. The p-value should be something like p=Pr{data|H0} where you'd need to define "Pr", "data" and "H0" carefully. (Pr is long-run relative frequency, data is statistic as observed or more extreme, H0 means the null is true and test assumptions hold). Also the power is computed under a specific alternative hypothesis. You can't leave it at "not H0". $\endgroup$
    – dipetkov
    Commented Dec 29, 2023 at 15:15
  • 2
    $\begingroup$ That being said, it may be interesting to read this blog by Daniel Lakens: The relation between p-values and the probability H0 is true is not weak enough to ban p-values and references therein. Note however that the post ends with "If you really want to make statements about the probability the null-hypothesis is true, given the data, p-values are not the tool of choice (Bayesian statistics is)." $\endgroup$
    – dipetkov
    Commented Dec 29, 2023 at 15:21
  • 4
    $\begingroup$ Your project seems doomed to fail for the following reasons. Testing a null addresses the question: Are the data reasonably consistent with the null hypothesis? A p-value is a measure of this consistency, and can be regarded as a measure of the absolute plausibility of the null. A different question is: Given the data and two competing hypotheses, what is the relative plausibility of each? Posterior probabilities are Bayesian answers for the relative plausibilty of question. $\endgroup$ Commented Dec 29, 2023 at 20:15
  • 2
    $\begingroup$ I think a great session for the clinicians would be to get them to write down their own prior for the example you have, then ask say 5 of them what their priors are, input those priors one at a time into the Bayes calculator to get 5 posteriors, and report the 5 Bayesian answers back to the class. Then have a discussion about the pros and cons of both methods. $\endgroup$ Commented Dec 30, 2023 at 18:46

1 Answer 1

1
$\begingroup$

Can I convert a frequentist p-value into a Bayesian Posterior Probability

No.

Bayes theorem is

$$P(H_0|\text{data}) = \frac{P(\text{data}|H_0)}{P(\text{data})} \cdot P(H_0)$$

as mentioned in the comments, this $P(\text{data}|H_0)$ is not the same as a p-value.


Related questions

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.