Let's say we want to find the posterior distribution for $\Theta$, where the likelihood model $X|\Theta$ ~ $Binom(8000, \Theta)$. Suppose instead of one distribution for the prior, we use a linear combination. For example $$f_\Theta(\theta) = \frac{1}{3}Beta(10, 1)+\frac{1}{3}Beta(1, 1)+\frac{1}{3}Beta(1, 10)$$

Since the beta distribution is the conjugate prior for the probability of the binomial distribution, the posterior I got is $$P_{\Theta|X}(\theta|x) = w_1*Beta(10+x, 8001-x)+w_2*Beta(1+x, 8001-x)+w_3*Beta(1+x, 8010-x)$$

It seems to make sense that a linear combination of priors results in a linear combination of posteriors, however, I am not sure how to update the weights. Intuitively, the more "likely" posteriors should have a larger weight. Is my approach on the right track?


You are correct to think that the (mixture) weights need updating. I'm going to answer your question a bit more generally than it was asked (and I'm going to change the notation in order to do that).

The problem can be expressed in terms of Bayes rule: \begin{equation} p(\theta|x) = \frac{p(x|\theta)\,p(\theta)}{p(x)} , \end{equation} where \begin{equation} p(x) = \int p(x|\theta)\,p(\theta)\,d\theta . \end{equation}

The thing of interest here is that the prior is a mixture \begin{equation} p(\theta) = \sum_j w_j\, p_j(\theta) . \end{equation} Given the mixture prior, here is some useful notation: \begin{equation} p_j(\theta|x) = \frac{p(x|\theta)\,p_j(\theta)}{p_j(x)} , \end{equation} where \begin{equation} p_j(x) = \int p(x|\theta)\,p_j(\theta)\,d\theta . \end{equation}

We can now express the solution: \begin{equation} p(\theta|x) = \frac{\sum_j w_j\,p(x|\theta)\,p_j(\theta)}{\int \sum_j w_j\,p(x|\theta)\,p_j(\theta)\,d\theta} = \frac{\sum_j w_j\,p_j(\theta|x)\,p_j(x)}{\sum_j w_j\,p_j(x)} = \sum_j \widetilde w_j\,p_j(\theta|x) , \end{equation} where \begin{equation} \widetilde w_j = \frac{w_j\,p_j(x)}{\sum_{j'} w_{j'}\,p_{j'}(x)} . \end{equation} The result is a mixture where the mixture weights have been updated (in addition to the component distributions).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.