For some time I have been reading into Bishop's Pattern Recognition and Machine Learning. Coming back to some earlier chapters the following got me confused and I am interested where, formally I go wrong.

Bishop states in chapter 1 that Bayes' Theorem \begin{equation} p(C\mid x) = \frac{p(x \mid C)p(C)}{p(x)} \end{equation} holds for probability densities (as well as probability masses and cont/discrete mixed densities). The core problem is at the meaning of the denominator $p(x)$. Is it an arbitrary prob. density or does the normalization constraint need to hold? What are the constraints for the choice of the RHS terms?

Consider the following example:

Assume $x\in \mathbb{R}$ to be a continuous variable and $C$ a class label $C \in \{0,1\}$.

As an example, could we imagine a situation with a marginal density given by a Gaussian \begin{equation} p(x) = N(x\mid \mu,\sigma) \\ \end{equation} and where at the same time the class-conditional probabilities (likelihood) are given by Gaussians \begin{align} p(x\mid C=0) &= N(x\mid \mu-\epsilon,\sigma) \\ &and\\ p(x\mid C=1) &= N(x\mid \mu+\epsilon,\sigma) \\ \end{align} that are shifted by $\delta$ from the mode of the marginal distribution?

Assuming equal priors $p(C=0)=p(C=1)=0.5$ I am tempted to simply evaluate the LHS of the first equation, e.g. for an $x$ on the far LHS of $\mu$ I find \begin{equation} p(C=0\mid x) = 0.5\frac{N(x\mid \mu -\epsilon,\sigma)}{N(x\mid \mu,\sigma)}=...\propto \exp(\frac{\delta(-x+(\mu-\epsilon/2)}{\sigma^2}) \end{equation} This expression diverges towards smaller $x$. We know that the posterior does not have to be normalized with respect to $x$, and also I presume it can be $>1$ as it is a probability density.

But, I think it still has to hold that $\sum_C p(C\mid x) = \sum_C \frac{p(x,C)}{p(x)} = \frac{\sum_C p(x,C)}{p(x)}=\frac{p(x)}{p(x)}=1$. Which cannot be with the posterior at hand (direct consequence of not choosing a normalizing marginal).

TL;DR: "Why" can is start with "apparently arbitrary" expressions for the RHS $p(x)$ and $p(x\mid C)$ and find an expression for the LHS, when it turns out that the densities are not consistent?


1 Answer 1


First, the identity$$\begin{equation}p(c\mid x) = \frac{p(x \mid c)p(c)}{p(x)} \end{equation}\tag{1}$$is not open to options or debate. It is a theorem, meaning that it holds when used with the proper definitions of the terms within it. (In continuous cases, it holds almost everywhere.) This means that, given a joint probability density $p(x,c)$ for the pair of rv's $(X,C)$, wrt a dominating measure, say $\text d\lambda(x)\text d\mu(c)$, the marginal densities$$p(x)=\int p(x,c)\text d\mu(c)\qquad p(c) = \int p(x,c)\text d\lambda(x)$$are (almost surely) uniquely defined. And thus that the conditional densities$$p(x|c)=\dfrac{p(x,c)}{p(c)}\qquad p(c|x)=\dfrac{p(x,c)}{p(x)}$$are also (almost surely) uniquely defined. The theorem (and theory) does not apply when some of the four elements in (1) are arbitrarily chosen.

Second, the example does not apply for the reason above: $p(x)$ as chosen is not compatible with $p(x|c)$ as chosen. If $X|C=c\sim\mathcal N(\mu+(-1)^c\epsilon,\sigma^2)$ and $\mathbb P(C=0)=\mathbb P(C=1)=1/2$, then $$X\sim 1/2\mathcal N(\mu+\epsilon,\sigma^2)+1/2\mathcal N\mathcal (\mu-\epsilon,\sigma^2)$$which is not a $\mathcal N(\mu,\sigma^2)$ distribution.

  • $\begingroup$ Thank you. Recap: Say the joint probabilty is known --> conditionals and marginals follow by definition --> all expressions in Bayes Theorem are defined, the theorem itself is provable consequence. Making an Ansatz for any term(s) in (1), the other terms have to be consistent (likely unique) w.r.t. definitions of marginal and conditional. Only then Bayes theorem (1) holds. @ Xi'an: In case you read this, a dashing question: Would you mind having a look at this post: stats.stackexchange.com/questions/406113/… $\endgroup$
    – bebi
    May 8, 2019 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.