I am reviewing my lectures in Machine Learning and my current topic is Bayesian Classifier. The context is the classification of two classes C1 and C2.

My book (neural networks and learning machines by haykin) states that the Bayesian classifier, under the assumption that the two classes have a multivariate Gaussian distribution and we are basing our decision on the log likelyhood ratio the classifier acts as a linear classifier.

However, looking on wikipedia it states that the Bayes classifier acts as a linear classifier only when classes have multinomial distribution.

I know that the multinomial distribution is similar to the multivariate gaussian for a high number of samples but I don't understand why wikipedia is not specifying the case described in my book. Am I missing something?


Start by writing the PDF of a multivariate gaussian

$$ p(x|C_{k}) = det(2\pi\Sigma)^{-\frac{1}{2}}e^{-\frac{1}{2}(\vec{x} - \vec{u})'\Sigma^{-1}(\vec{x} - \vec{u})} $$

Then take the logarithm of both sides to get rid of the exponent $$ ln(p(x|C_{k})) = ln(det(2\pi\Sigma)^{-\frac{1}{2}}) + -\frac{1}{2}(\vec{x} - \vec{u})'\Sigma^{-1}(\vec{x} - \vec{u}) $$

Next use Bayes rule $$ ln(p(C_{k}|x)) \propto ln(p(C_{k})) + ln(p(x|C_{k})) $$ Then substitute into Bayes rule for $ln(p(x|C_{k}))$ $$ ln(p(C_{k}|x)) \propto ln(p(C_{k})) + ln(det(2\pi\Sigma)^{-\frac{1}{2}}) -\frac{1}{2}(\vec{x} - \vec{u})'\Sigma^{-1}(\vec{x} - \vec{u}) $$ And combine all the constants into $b$ $$ ln(p(C_{k}|x)) \propto - \frac{1}{2}(\vec{x} - \vec{u})'\Sigma^{-1}(\vec{x} - \vec{u}) + b $$

and you can see all that remains is a constant addition plus matrix multiplication, all of which are linear operations. So in the log space for a Multivariate Gaussian distributed class, a Bayes classifier is a linear classifier :).

  • $\begingroup$ Thanks for the explanation! So I guess Wikipedia is just missing to state that also for Multivariate Gaussian the Bayes classifier reduces to linear? $\endgroup$ – Iacopo Olivo Feb 21 at 5:40
  • $\begingroup$ Yes, show a similar proof for any class distribution from the exponential family of distributions. $\endgroup$ – Brent Feb 21 at 18:50
  • $\begingroup$ Ok, that solves all my doubts. Thank you for your help! $\endgroup$ – Iacopo Olivo Feb 22 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.