# Why do the posterior probabilities violate the axioms of probability when we apply Bayesian update without likelihood computation?

Suppose that the unknown parameter $$\Theta$$ is Bernoulli and we make $$n$$ observations $$X_1,X_2,\ldots,X_n$$, which are continuous random variables. Assuming that $$X_1,X_2,\ldots,X_n$$ are conditionally independent given $$\Theta$$, Bayes rule is

$$p_{\Theta|X_1,X_2,\ldots,X_n}(\theta|x_1,x_2,\ldots,x_n)=\frac{p_\Theta(\theta)\prod_{i=1}^nf_{X_i|\Theta}(x_i|\theta)}{\sum_{\theta'=0}^1p_\Theta(\theta')\prod_{i=1}^nf_{X_i|\Theta}(x_i|\theta')}\tag1\label1$$

Assuming that $$X_1,X_2,\ldots,X_n$$ are unconditionally independent, Bayes rule is

$$p_{\Theta|X_1,X_2,\ldots,X_n}(\theta|x_1,x_2,\ldots,x_n)=\frac{p_\Theta(\theta)\prod_{i=1}^nf_{X_i|\Theta}(x_i|\theta)}{\prod_{i=1}^nf_{X_i}(x_i)}\tag2\label2$$

Substituting $$\frac{f_{X_i|\Theta}(x_i|\theta)}{f_{X_i}(x_i)}=\frac{p_{\Theta|X_i}(\theta|x_i)}{p_\Theta(\theta)}$$, we have

$$p_{\Theta|X_1,X_2,\ldots,X_n}(\theta|x_1,x_2,\ldots,x_n)=\frac{\prod_{i=1}^n p_{\Theta|X_i}(\theta|x_i)}{(p_\Theta(\theta))^{n-1}}\tag3\label3$$

This form is used to implement a Bayes classifier, where $$\Theta$$ is the target and $$X_i$$ are the features. The numerator terms are approximated by using the proportion of radius neighbors (i.e. points falling within a radius). For example, if 100 points fall within the radius of $$X_i$$ and 10 of them have $$\Theta=1$$, then we assign $$p_{\Theta|X_i}(1|x_i)=0.1$$. If there are too few or no neighbors, we assign $$p_{\Theta|X_i}(\theta|x_i)=p_\Theta(\theta)$$.

This method has a problem. If $$p_{\Theta|X_i}(\theta|x_i)\gg p_\Theta(\theta)$$ for all $$i$$, then $$p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)>1$$ and $$p_{\Theta|X_1,X_2,\ldots,X_n}(0|x_1,x_2,\ldots,x_n)+p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)\ne1$$. Thus, two of the axioms of probability are violated.

Why does this method have this problem? We can remedy that by normalizing the posterior probabilities, i.e. dividing by $$p_{\Theta|X_1,X_2,\ldots,X_n}(0|x_1,x_2,\ldots,x_n)+p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)$$. However, this feels like hacking. The model described performs well in practice, but I would like a more careful analysis of this approach.

• Kolmogorov's 2nd axiom $P(\Omega)=1$ is unsatisfied, this is why you need a normalisation constant (marginal likelihood). This isn't hacking, it's just applying the math. You will not have a posterior probability unless you normalise. – Digio Mar 28 at 12:26
• @Digio Bayes rule already has a normalization constant. If we specify a likelihood model and use $\ref1$, the output is already normalized. However, if we use $\ref3$ by directly approximating the posterior given each feature, the output may be unnormalized. How can a formula already having a normalization constant produce an unnormalized output? – W. Zhu Mar 28 at 13:08
• I have not looked at this in detail, but I think there may be a problem with your assumption that the observations are both conditionally and unconditionally independent. If you look at some concrete situations where this occurs, you may find your problem. – guy Mar 28 at 13:44
• OK, now I get what you mean. But isn't this how Variational Bayes and EM work? You update your posterior distribution based on the current iteration's MAP/MLE in a sequential fashion until convergence. – Digio Mar 28 at 14:56
• Are you sure that $p(\theta|X) = \frac{p(\theta|X)}{p(\theta)}$..? – Tim Mar 28 at 21:39