9
$\begingroup$

There are a lot of places where you'll see the proof that Naive Bayes classifiers are linear, like this and this. But they always assume a special case of the family of Naive Bayes classifiers which more often than not happens to be Multinomial Naive Bayes.

$$ P(X_1 = x_1, X_2 = x_2, \cdots , X_k = x_k) = \frac{n!}{x_1!x_2!\cdots x_k!} p_1^{x_1} p_2^{x_2} \cdots p_k^{x_k} $$

Now, Multinomial Naive Bayes can be proved to be a linear classifier as follows:

Considering a binary classification example with $y \in \{0, 1\} $, the decision boundary would be:

$$ \begin{equation} log(\frac{P(Y = 1|X = x)}{P(Y = 0|X = x)}) = 0 \end{equation} $$

where $X = (X_1, X_2, \cdots ,X_k)$ and $x = (x_1, x_2, \cdots ,x_k)$

Using Bayes' rule and multinomial distribution, the decision boundary could be written as:

$$ \sum_{i=1}^{k} x_i log(\frac{p_i}{p_i^{'}}) + log(\frac{P(Y=1)}{P(Y=0)}) = 0 $$

where $p_i$ and $p_i^{'}$ are probabilities of the occurrence of ith feature in class $y = 1$ and $y = 0$ respectively.

So, we've got a linear form ($w^Tx + b = 0$) for our decision boundary.

But is this true for the entire family of Naive Bayes classifiers?

I suspect it would depend on the the nature of the probability distributions you choose for the input features.

$\endgroup$
1
  • 1
    $\begingroup$ I was thinking about this recently. I wrote a related answer earlier (on naive bayes and discriminant analysis methods) -- stats.stackexchange.com/a/297877/1704 -- but I didn't look into whether all the "Discriminant Analysis" methods were linear. I know that QDA is not linear. And, as you mentioned, the standard discrete/multinomial NB is linear). Need to think about the others. A quick experiement had GNB (gaussian NB) non-linear, but I'm not convinced I didn't make a mistake. $\endgroup$
    – MrDrFenner
    Mar 27, 2018 at 12:28

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.