In this tutorial on Naive Bayes Classififer in section 1.1, the author proved naive bayes is a linear classifier.

Consider binary classification where $y=0$ or $1$. Our classification rule with argmax can equivalently be expressed with log odds ratio $$f(x)=log\ \dfrac{p(y=1|x)}{p(y=0|x)}\\=log\ p(y=1|x)-log\ p(y=0|x)\\=(log\ \theta_1-log\ \theta_0)^\top x+(log\ p(y=1)-log\ p(y=0))$$ The decision rule is to classify $x$ with $y=1$ if $f(x)>0$, and $y=0$ otherwise. Note for given parameters, this is a linear function in $x$. That is to say, the Naive Bayes classifier induces a linear decision boundary in feature space $X$ . The boundary takes the form of a hyperplane, defined by $f(x) = 0$.

Can you tell me how is the third line of equation deducted from the second line? It doesn't seem clear to me.


1 Answer 1


In section 1, the log likelihood is given: $\log p(x|y) = x^T \log \theta_y + const$ as well as Bayes rule: $p(y|x) = \frac{p(x|y)p(y)}{p(x)}$, then it should be something like:

$$\log p(y=1|x) - \log p(y=0|x) \\ = \log \frac{p(x|y=1)p(y=1)}{p(x)} - \log \frac{p(x|y=0)p(y=0)}{p(x)} \\ = \log p(x|y=1)p(y=1) - \log p(x|y=0)p(y=0) \\ = (x^T \log \theta_1 + \log p(y=1)) - (x^T \log \theta_0 + \log p(y=0)) \\ = (\log \theta_1 - \log \theta_0)^T x + (\log p(y=1) - \log p(y=0))$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.