2
$\begingroup$

Given the $k$th Gaussian distribution $N \sim (\mu_k, \Sigma_k)$, the probability that $x_i$ generated from this Gaussian $k$ can be found via Bayes' rule $$\begin{align}p(z_i = k | x_i,\mu_k, \Sigma_k) &= \frac{p(x_i,z_i =k)}{p(x)} \\ &= \frac{\pi_kN(x_i|\mu_k,\Sigma_k)}{\sum_{k=1}^m\pi_kN(x_k|\mu_k,\Sigma_k)}\end{align}$$ where $p(x,z_i=k)$ is the joint probability density distribution while $p(x)$ is the marginal distribution over the mixture of Gaussians.

Bayes' theorem in machine learning is applied in the following manner, when estimating the posterior of the model parameters $\theta$, $$p(\theta|D) = \frac{p(\theta)p(D|\theta)}{\int p(D|\theta)p(\theta)d\theta}$$ In this case $p(D|\theta)$ is a conditional probability because $\theta$ is a random variable.

  1. why is it the case that $N(x_i|\mu_k,\Sigma_k)$ is not a conditional probability but can still be used in Bayes' theorem ?
  2. Is the numerator in Bayes' theorem a distribution or a discrete probability? When is it the case where it is a distribution and when is it the case where the numerator is a probability. I know that $p(\theta)p(D|\theta)$ is a distribution over $\theta$ and $\pi_kN(x_i|\mu_k,\Sigma_k)$ is also the joint distribution.
$\endgroup$
1
  • $\begingroup$ I corrected the first formula as it is a probability conditional on $X_i=x_i$. $\endgroup$
    – Xi'an
    Aug 6, 2020 at 14:28

1 Answer 1

2
$\begingroup$
  1. why is it the case that $N(x_i|\mu_k,\Sigma_k)$ is not a conditional probability but can still be used in Bayes' theorem ?

$N(x_i|\mu_k,\Sigma_k)$ is a conditional probability density function. It is conditional on cluster assignment $z_i = k$. High-level formula is

$$ p(k | X) \propto p(X|k)\, p(k) $$

  1. Is the numerator in Bayes' theorem a distribution or a discrete probability? When is it the case where it is a distribution and when is it the case where the numerator is a probability. I know that $p(\theta)p(D|\theta)$ is a distribution over $\theta$ and $\pi_kN(x_i|\mu_k,\Sigma_k)$ is also the joint distribution.

$p(z_i=k) = \pi_k$ is a Bernoulli distribution (discrete), while $p(x_i|z_i=k) = N(x_i|\mu_k,\Sigma_k)$ is a probability density function, and the result is a probability density.

$\endgroup$
3
  • $\begingroup$ Bayes theorem for discrete case has likelihood term $p(X|k)$ conditioned on probability whereas for the continuous variable case it is a pdf ? $\endgroup$
    – calveeen
    Aug 6, 2020 at 11:15
  • $\begingroup$ @calveeen en.wikipedia.org/wiki/Bayes%27_theorem#Random_variables $\endgroup$
    – Tim
    Aug 6, 2020 at 11:28
  • $\begingroup$ I see thank you >.< $\endgroup$
    – calveeen
    Aug 6, 2020 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.