5
$\begingroup$

I think that I miss something very basic about the posterior distributions in bayesian hypothesis testing. In the frequentist approach, we define the null hypothesis, compute the test power, define an rejection threshold, collect the data and if the p-value is below the rejection threshold, we reject the null hypothesis.

What happens in the Bayesian case? Let's say we compare the conversion rates of two versions of a web site. We define the priors, obtain the data and get two posterior distributions, one for each version of the site. For example, the distributions look like this two slightly overlapping PDF curves

What now? What can we say about the two versions? Do we "reject" one curve? Is the are common to the two curves relevant to any analysis, and if it is, what does it mean?

$\endgroup$
6
$\begingroup$

What can we say about the two versions?

You are directly modelling the probability distributions for the two conversion rates. You can use the posterior to answer questions about the two conversion rates. Such questions might be...

  • What is the probability the new version has a larger conversion rate?

  • What is the probability the new version has a smaller conversion rate?

etc.

You can answer whatever question your posterior allows you to answer.

Do we "reject" one curve?

No. Like I said, you are modelling the posterior distribution. You can use the posterior to answer any hypotheses you've made.

$\endgroup$
  • $\begingroup$ Thanks. Does the overlapping area have any meaningful interpretation? $\endgroup$ – Ivan Sanych Jul 18 '19 at 6:38
  • $\begingroup$ I can't say for sure. $\endgroup$ – Demetri Pananos Jul 18 '19 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.