# Posterior probability of hypothesis distributions

Suppose I have $$K$$ classes with distribution $$\theta$$ over $$\{1,...,K\}$$ and an underlying domain $$D$$ on which each class defines a categorical distribution $$\phi_i$$.

Given a draw $$i\sim\theta$$ and $$x\sim\phi_i$$, where only $$x$$ is observed, I want to update both $$\theta$$ and the $$\phi_i$$'s. The posterior for $$\theta$$ is easy:

$$\hat{\theta}(i) = P(i\mid x) \propto P(x\mid i)\cdot P(i) = \phi_i(x)\cdot \theta(i)$$

Is it possible to compute a posterior for $$\phi_i$$ as well? From the definition of the posterior, it seems that it should be:

$$\hat{\phi}_i(\tilde{x}) = P(\tilde{x}\mid x, i) \propto P(x\mid \tilde{x}, i)\cdot P(\tilde{x} \mid i) = \phi_i(x) \cdot \phi_i(\tilde{x})$$.

but something about that just seems wrong. Shouldn't $$\theta$$ appear in the numerator somewhere? Am I interpreting the likelihood term wrong?

You seem to be talking about posterior predictive distribution, i.e. the a posteriori distribution of the data. You don't see $$\theta$$, because it is marginalized over possible parameter values. The distribution of the data given some particular parameter value is the likelihood function $$P(x|i) = \phi_i(x)$$.

Regarding your comments, I guess the other thing that you may mean is Bayesian updating. Given some data we update a prior

$$p(\theta | x) \propto p(x | \theta)\,p(\theta)$$

next, knowing this you can use the posterior as a prior to be updated with new data $$\tilde{x}$$,

$$p(\theta|x,\tilde{x}) \propto p(\tilde{x}|\theta,x) \, p(\theta|x)$$

By the chain rule, this can done in one step. So you just plug-in the posterior estimates given $$x$$ as a prior for the likelihood for $$\tilde{x}$$, this may be what you are asking about.

• Yes, I see how the posterior predictive is related, but would still like to condition on $i$ so that I can use it to update the $\phi_i$ parameter. It seems what I want to do is update the likelihood function. Perhaps that's not possible or I need more hyperparameters?
– Bill
Commented Feb 11, 2020 at 15:59
• @Bill likelihood is conditioned on $i$, so I'm not sure what you mean?
– Tim
Commented Feb 11, 2020 at 16:24
• right, but $P(\tilde{x}\mid i)$ is just $\phi_i(\tilde{x})$. I want to update the $\phi_i$'s with something like $\hat{\phi}_i(\tilde{x}) = P(\tilde{x}\mid x,i)$, which I guess is the posterior predictive conditioned on $i$?
– Bill
Commented Feb 11, 2020 at 16:38
• @Bill I can't see how would this differ from likelihood?
– Tim
Commented Feb 11, 2020 at 17:10

The model considered there is a mixture $$X\sim\sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x)$$ The posterior distribution on the parameters of the model is thus $$\pi(\theta,\phi|x_1,\ldots,x_n)\propto\pi(\theta,\phi)\prod_{k=1}^n \sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x_k)$$ with no manageable closed form expression for the marginal posteriors $$\pi(\theta|x_1,\ldots,x_n)$$ and $$\pi(\phi|x_1,\ldots,x_n)$$ or even the conditional posteriors $$\pi(\theta|x_1,\ldots,x_n,\phi)$$ and $$\pi(\phi|x_1,\ldots,x_n,\theta)$$ (except when $$n$$ is very small).