I know there are a lot of questions here about ignoring the denominator in a Bayesian approach, but I don't think mine is a duplicate of any of them.

I am reading the book "Pattern recognition and machine learning" by Cristopher Bishop.

Imagine we have a set of N observations of a (single) variable, which we collect in a vector $\mathbf{x} \in \mathcal{R}^N$. We would like to find the mean $\mu$ of the probbility density function that generated that data, using a Bayesian approach. Thus, we first need to find the posterior probability $p(\mu|\mathbf{x})$

We can write:

$p(\mu|\mathbf{x}) = p(\mathbf{x}|\mu) \cdot \dfrac{p(\mu)}{p(\mathbf{x})}$

Now as the book says, we can ignore the denominator because it is just a normalizing factor

$p(\mu|\mathbf{x}) \propto p(\mathbf{x}|\mu) \cdot p(\mu) = p(\mathbf{x}, \mu) $

where the last equation follows from the product rule, or the defition of conditional density for $p(\mathbf{x}|\mu)$ if you want.

So we are approximating a conditional distribution with a joint distribution? How is that even possible?

For one, $p(\mu|\mathbf{x})$ whould be a function of $\mathbf{x}$, while $p(\mathbf{x}, \mu) $ whould be a function of both $\mathbf{x}$ and $\mu$, right?


First of all, $p(\mu|\mathbf{x})$ is not only a function of $\mathbf{x}$. It is again a function of both $\mu$ and $\mathbf{x}$, as joint PDF is. Your question would make sense if we were using $p(\mathbf{x})$, i.e. a function of only $\mathbf{x}$, instead of the joint, which we don't do of course.

Another thing is, we don't approximate the conditional PDF with joint PDF. The $\mu$ that maximizes the joint, also maximizes the conditional. This is just MAP estimation, where we choose $\mu$ such that the posterior, i.e. $p(\mu|\mathbf{x})$, is maximized (this also means that $p(\mu|\mathbf{x})$ is not only a function of $\mathbf{x}$, but also $\mu$). There, you can ignore the denominator since it doesn't depend on $\mu$, and acts as a scalar for the specific optimization problem.

There might be cases where you don't ignore the denominator. For example, conditional mean, i.e. $E[\mu|\mathbf{x}]$, sometimes called Bayesian Parameter Estimation in which you generally need to explicitly find $p(\mu|\mathbf{x})$, (especially if it's not in a common format) and calculate the conditional mean.

| cite | improve this answer | |
  • $\begingroup$ ok, so the point is that the mode of $p(\mu|x)$ equals the mode of $p(\mu, x)$? $\endgroup$ – raffaem Apr 27 '19 at 10:57
  • 1
    $\begingroup$ If you take $\mathbf{x}$ as constant, i.e. a slice of $p(\mu,\mathbf{x})$, like you do in parameter estimation; yes, they have the same mode. Normally, you wouldn't say that they have the same mode since the latter is a multivariate function, and its mode is of the form $(\mu,\mathbf{x})$, i.e. both variables are changing. $\endgroup$ – gunes Apr 27 '19 at 11:03
  • $\begingroup$ didn't we establish that $p(\mu|x)$ is also a function of both $\mu$ and $\mathbf{x}$ too? $\endgroup$ – raffaem Apr 27 '19 at 12:47
  • 1
    $\begingroup$ Expression of $p(\mu|\mathbf{x})$ includes both $\mu$ and $\mathbf{x}$, but in this context, $\mathbf{x}$ is assumed to be given, so constant. Its mode is for $\mu$, and equal to $p(\mathbf{x},\mu)$'s mode with the same $\mathbf{x}$. $\endgroup$ – gunes Apr 27 '19 at 13:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.