Suppose that the unknown parameter $\Theta$ is Bernoulli and we make $n$ observations $X_1,X_2,\ldots,X_n$, which are continuous random variables. Assuming that $X_1,X_2,\ldots,X_n$ are conditionally independent given $\Theta$, Bayes rule is


Assuming that $X_1,X_2,\ldots,X_n$ are unconditionally independent, Bayes rule is


Substituting $\frac{f_{X_i|\Theta}(x_i|\theta)}{f_{X_i}(x_i)}=\frac{p_{\Theta|X_i}(\theta|x_i)}{p_\Theta(\theta)}$, we have

$$p_{\Theta|X_1,X_2,\ldots,X_n}(\theta|x_1,x_2,\ldots,x_n)=\frac{\prod_{i=1}^n p_{\Theta|X_i}(\theta|x_i)}{(p_\Theta(\theta))^{n-1}}\tag3\label3$$

This form is used to implement a Bayes classifier, where $\Theta$ is the target and $X_i$ are the features. The numerator terms are approximated by using the proportion of radius neighbors (i.e. points falling within a radius). For example, if 100 points fall within the radius of $X_i$ and 10 of them have $\Theta=1$, then we assign $p_{\Theta|X_i}(1|x_i)=0.1$. If there are too few or no neighbors, we assign $p_{\Theta|X_i}(\theta|x_i)=p_\Theta(\theta)$.

This method has a problem. If $p_{\Theta|X_i}(\theta|x_i)\gg p_\Theta(\theta)$ for all $i$, then $p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)>1$ and $p_{\Theta|X_1,X_2,\ldots,X_n}(0|x_1,x_2,\ldots,x_n)+p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)\ne1$. Thus, two of the axioms of probability are violated.

Why does this method have this problem? We can remedy that by normalizing the posterior probabilities, i.e. dividing by $p_{\Theta|X_1,X_2,\ldots,X_n}(0|x_1,x_2,\ldots,x_n)+p_{\Theta|X_1,X_2,\ldots,X_n}(1|x_1,x_2,\ldots,x_n)$. However, this feels like hacking. The model described performs well in practice, but I would like a more careful analysis of this approach.

  • 2
    $\begingroup$ Kolmogorov's 2nd axiom $P(\Omega)=1$ is unsatisfied, this is why you need a normalisation constant (marginal likelihood). This isn't hacking, it's just applying the math. You will not have a posterior probability unless you normalise. $\endgroup$ – Digio Mar 28 at 12:26
  • $\begingroup$ @Digio Bayes rule already has a normalization constant. If we specify a likelihood model and use $\ref1$, the output is already normalized. However, if we use $\ref3$ by directly approximating the posterior given each feature, the output may be unnormalized. How can a formula already having a normalization constant produce an unnormalized output? $\endgroup$ – W. Zhu Mar 28 at 13:08
  • $\begingroup$ I have not looked at this in detail, but I think there may be a problem with your assumption that the observations are both conditionally and unconditionally independent. If you look at some concrete situations where this occurs, you may find your problem. $\endgroup$ – guy Mar 28 at 13:44
  • $\begingroup$ OK, now I get what you mean. But isn't this how Variational Bayes and EM work? You update your posterior distribution based on the current iteration's MAP/MLE in a sequential fashion until convergence. $\endgroup$ – Digio Mar 28 at 14:56
  • $\begingroup$ Are you sure that $p(\theta|X) = \frac{p(\theta|X)}{p(\theta)}$..? $\endgroup$ – Tim Mar 28 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.