1
$\begingroup$

Let's assume we have prior distribution beta with parameters 2,50. Let's just say it's prior knowledge of sign up rates for our product.

Then we have two binomial models A and B, which both samples from the same prior. The population is 100 and the target for A is 2, while the target for B is 20.

Now, if you check posterior distribution in your preferred tool, you can see model B has much better rates. The distribution curve of model B is shifted much more to the right of the X-axis than for model A. Does this mean there are much bigger chances for 20 people to sign up for our product than 2??? This seems very unlogical, considering our prior distribution. How should I interpret this? Thank you.

Improve this question
$\endgroup$
2
  • $\begingroup$ "Then we have two binomial models A and B, which both samples from the same prior" How does a model sample from a prior distribution? $\endgroup$
    – TrungDung
    Commented Dec 9, 2019 at 14:24
  • $\begingroup$ What does "the target for A" mean? $\endgroup$
    – jbowman
    Commented Dec 10, 2019 at 20:48

1 Answer 1

Reset to default
1
$\begingroup$

To be clear - your prior assumes that you’ve observed ($2-1 = 1$) successes and ($50-1 = 49$) failures for an effective sample size of $N_1=50$. The posterior does not sample from the prior but rather combines the likelihood with the prior to arrive at the posterior distribution over $\theta$ (conversion rate in your case). Since the posterior mean is as a weighted average between the prior and observed means, the corresponding means of model A and model B are:

$\hat{\theta}_A=\frac{N_1}{(N_1+N_2)}\cdot\frac{\alpha}{\alpha+\beta} + \frac{N_2}{(N_1+N_2)}\cdot\frac{\alpha+n_A}{\alpha+n_A+\beta+(100-n_A)}$

$\hat{\theta}_B=\frac{N_1}{(N_1+N_2)}\cdot\frac{\alpha}{\alpha+\beta} + \frac{N_2}{(N_1+N_2)}\cdot\frac{\alpha+n_B}{\alpha+n_B+\beta+(100-n_B)}$

Where $N_1$ is the effective sample size of the prior, $N_2$ is the number of new observations, $n_A$ is the count of successes observed for model A, and $n_B$ is the count of successes observed for model B

As you can see, your results are not illogical - you are observing that the posterior is overwhelming the prior in the case of model B (double the data, 7x the conversion / success rate):

$\hat{\theta}_B=\frac{50}{150}\cdot\frac{1}{50} + \frac{100}{150}\cdot\frac{21}{150}$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much. I am still a bit confused if I am honest. If we just sample from prior and generate from binomial of population 100 and we check the histogram, we can see there is most likely 3 people will engage on signing up the product and almost no chance 20 people will do it. This is why I would expect posterior would be in favour of model A and would be shifted right on X axis, while for model B would be shefited left as there are almost no chances 20 people would sign up. Do you understand my point? $\endgroup$
    – Stenga
    Commented Dec 9, 2019 at 17:18
  • $\begingroup$ If by "most likely 3 people", you mean "the expected value is 3" then you are mis-specifying the prior. If you sample from your prior, both models should give you an expected value of 2 since $\hat{p}=.02$, $n=100$ and $np=(100)(.02)=2$ so you should "expect" 2 people to engage. I don't understand your point - could you clarify what you mean when you say "the target for A is 2 and the target for B is 20"? I was assuming that these are new observations that you are using to update your prior. If these are in fact new observations, then (again) your result is exactly what you should expect. $\endgroup$
    – cdlm
    Commented Dec 10, 2019 at 17:46
  • $\begingroup$ Yes, you are correct. I think I understand now. $\endgroup$
    – Stenga
    Commented Dec 11, 2019 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.