Suppose I have $K$ classes with distribution $\theta$ over $\{1,...,K\}$ and an underlying domain $D$ on which each class defines a categorical distribution $\phi_i$.

Given a draw $i\sim\theta$ and $x\sim\phi_i$, where only $x$ is observed, I want to update both $\theta$ and the $\phi_i$'s. The posterior for $\theta$ is easy:

$\hat{\theta}(i) = P(i\mid x) \propto P(x\mid i)\cdot P(i) = \phi_i(x)\cdot \theta(i)$

Is it possible to compute a posterior for $\phi_i$ as well? From the definition of the posterior, it seems that it should be:

$\hat{\phi}_i(\tilde{x}) = P(\tilde{x}\mid x, i) \propto P(x\mid \tilde{x}, i)\cdot P(\tilde{x} \mid i) = \phi_i(x) \cdot \phi_i(\tilde{x})$.

but something about that just seems wrong. Shouldn't $\theta$ appear in the numerator somewhere? Am I interpreting the likelihood term wrong?


2 Answers 2


You seem to be talking about posterior predictive distribution, i.e. the a posteriori distribution of the data. You don't see $\theta$, because it is marginalized over possible parameter values. The distribution of the data given some particular parameter value is the likelihood function $P(x|i) = \phi_i(x)$.

Regarding your comments, I guess the other thing that you may mean is Bayesian updating. Given some data we update a prior

$$ p(\theta | x) \propto p(x | \theta)\,p(\theta) $$

next, knowing this you can use the posterior as a prior to be updated with new data $\tilde{x}$,

$$ p(\theta|x,\tilde{x}) \propto p(\tilde{x}|\theta,x) \, p(\theta|x) $$

By the chain rule, this can done in one step. So you just plug-in the posterior estimates given $x$ as a prior for the likelihood for $\tilde{x}$, this may be what you are asking about.

  • $\begingroup$ Yes, I see how the posterior predictive is related, but would still like to condition on $i$ so that I can use it to update the $\phi_i$ parameter. It seems what I want to do is update the likelihood function. Perhaps that's not possible or I need more hyperparameters? $\endgroup$
    – Bill
    Commented Feb 11, 2020 at 15:59
  • $\begingroup$ @Bill likelihood is conditioned on $i$, so I'm not sure what you mean? $\endgroup$
    – Tim
    Commented Feb 11, 2020 at 16:24
  • $\begingroup$ right, but $P(\tilde{x}\mid i)$ is just $\phi_i(\tilde{x})$. I want to update the $\phi_i$'s with something like $\hat{\phi}_i(\tilde{x}) = P(\tilde{x}\mid x,i)$, which I guess is the posterior predictive conditioned on $i$? $\endgroup$
    – Bill
    Commented Feb 11, 2020 at 16:38
  • $\begingroup$ @Bill I can't see how would this differ from likelihood? $\endgroup$
    – Tim
    Commented Feb 11, 2020 at 17:10

The model considered there is a mixture $$X\sim\sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x)$$ The posterior distribution on the parameters of the model is thus $$\pi(\theta,\phi|x_1,\ldots,x_n)\propto\pi(\theta,\phi)\prod_{k=1}^n \sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x_k)$$ with no manageable closed form expression for the marginal posteriors $\pi(\theta|x_1,\ldots,x_n)$ and $\pi(\phi|x_1,\ldots,x_n)$ or even the conditional posteriors $\pi(\theta|x_1,\ldots,x_n,\phi)$ and $\pi(\phi|x_1,\ldots,x_n,\theta)$ (except when $n$ is very small).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.