# How do I determine my Bayesian sampling size when comparing two proportions?

I am currently working on writing a simulation in R to compare the results of Frequentist vs Bayesian when it comes to two-proportion hypothesis testing. For the Bayesian side, I am simply using a Beta-Binomial model for each of the proportions. Below is my code for determining the posterior distribution of the difference between the two samples:

postdist = rbeta(mcmcsize, prior1+treatment, prior2+tsamp-treatment) -
rbeta(mcmcsize, prior1+control, prior2+csamp-control)


My questions basically comes down to determining the value of 'mcmcsize'. I know I need to go large, but how do I determine what's large enough? Can I get too large?

Thanks.

• If you are using the Beta-Binomial model, then your posterior distribution is known (it is a Beta, right?). So you can draw exact samples from a beta distribution, which means you don't need MCMC. So what is your question? Commented Dec 8, 2016 at 21:42
• @Greenparker The quantity of interest, the difference between the two Beta distributions, is not Beta distributed.
– Emre
Commented Jun 26, 2020 at 9:01

The random MCMC sample (of the difference of parameter values) is merely a high-resolution picture of the posterior distribution. The bigger the sample, the better is the picture.

It can't ever become too high resolution in the sense of becoming inaccurate, so make your MCMC sample as big as you like. The only constraint is computer memory and processing.

But how big is big enough? A larger MCMC sample will provide a smoother picture of the distribution, but there is always some degree of random "wobble" in the sample no matter how big. If your goal is to have a reasonably stable estimate of a credible interval such as a 95% highest density interval, then you need a large sample to get stable estimates of the tail densities. In DBDA2E (pp. 182-187) I recommend an effective sample size (ESS) of 10,000. ESS is different than the number of steps in an MCMC chain, but in your specific case rbeta generates independent steps and mcmcsize is essentially the same as ESS.

Hope that answers the question you meant...

The beta-binomial is a conjugate model, so there is an exact analytic solution. There is no need for posterior simulation. If the prior is beta distributed and the likelihood is binomial, then the posterior is beta distributed. So, for example, if your prior distribution is $$\frac{p^{\alpha-1}(1-p)^{\beta-1}}{B(\alpha,\beta)}$$ and you had a sample size of $$n$$ and observed $$k$$ successes then your posterior density is $$\frac{p^{k+\alpha-1}(1-p)^{n-k+\beta-1}}{B(\alpha+k,n-k+\beta)}.$$

If your model were not conjugate, then you should sample until your parameters are burned in and until you have reached convergence. I have had that happen in 5,000 iterations and I have had that happen in 2,000,000 iterations. You are done when you are done.

It can also depend on why you are doing MCMC. If you are estimating the denominator of Bayes rule, then you are done when you are done, but if you are graphically presenting the posterior, then you are done when the graph is smooth enough for your presentation purposes.