1
$\begingroup$

Given the $k$th Gaussian distribution $N \sim (\mu_k, \Sigma_k)$, the probability that $x_i$ generated from this Gaussian $k$ can be found via Bayes' rule $$\begin{align}p(z_i = k | x_i,\mu_k, \Sigma_k) &= \frac{p(x_i,z_i =k)}{p(x)} \\ &= \frac{\pi_kN(x_i|\mu_k,\Sigma_k)}{\sum_{k=1}^m\pi_kN(x_k|\mu_k,\Sigma_k)}\end{align}$$ where $p(x,z_i=k)$ is the joint probability density distribution while $p(x)$ is the marginal distribution over the mixture of Gaussians.

Bayes' theorem in machine learning is applied in the following manner, when estimating the posterior of the model parameters $\theta$, $$p(\theta|D) = \frac{p(\theta)p(D|\theta)}{\int p(D|\theta)p(\theta)d\theta}$$ In this case $p(D|\theta)$ is a conditional probability because $\theta$ is a random variable.

  1. why is it the case that $N(x_i|\mu_k,\Sigma_k)$ is not a conditional probability but can still be used in Bayes' theorem ?
  2. Is the numerator in Bayes' theorem a distribution or a discrete probability? When is it the case where it is a distribution and when is it the case where the numerator is a probability. I know that $p(\theta)p(D|\theta)$ is a distribution over $\theta$ and $\pi_kN(x_i|\mu_k,\Sigma_k)$ is also the joint distribution.
$\endgroup$
  • $\begingroup$ I corrected the first formula as it is a probability conditional on $X_i=x_i$. $\endgroup$ – Xi'an Aug 6 at 14:28
2
$\begingroup$
  1. why is it the case that $N(x_i|\mu_k,\Sigma_k)$ is not a conditional probability but can still be used in Bayes' theorem ?

$N(x_i|\mu_k,\Sigma_k)$ is a conditional probability density function. It is conditional on cluster assignment $z_i = k$. High-level formula is

$$ p(k | X) \propto p(X|k)\, p(k) $$

  1. Is the numerator in Bayes' theorem a distribution or a discrete probability? When is it the case where it is a distribution and when is it the case where the numerator is a probability. I know that $p(\theta)p(D|\theta)$ is a distribution over $\theta$ and $\pi_kN(x_i|\mu_k,\Sigma_k)$ is also the joint distribution.

$p(z_i=k) = \pi_k$ is a Bernoulli distribution (discrete), while $p(x_i|z_i=k) = N(x_i|\mu_k,\Sigma_k)$ is a probability density function, and the result is a probability density.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Bayes theorem for discrete case has likelihood term $p(X|k)$ conditioned on probability whereas for the continuous variable case it is a pdf ? $\endgroup$ – calveeen Aug 6 at 11:15
  • $\begingroup$ @calveeen en.wikipedia.org/wiki/Bayes%27_theorem#Random_variables $\endgroup$ – Tim Aug 6 at 11:28
  • $\begingroup$ I see thank you >.< $\endgroup$ – calveeen Aug 6 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.