I understand the basic principles for the naïve bayes classification with one feature:

$$ P(Class|feature) = (P(f|Class) * P(Class)) / P(f) $$

We have a dataset that has the following attributes/features:

day | outlook | temperature | humidity | wind | play

  • Day is just a number (sequence)
  • Outlook can be [sunny | overcast | rain]
  • Temperature can be [cool | mild | hot]
  • Humidity can be [normal | high]
  • Wind can be [strong | weak]
  • Play is [yes | no]

Now, we have a new instance: today = (sunny, cool, high, strong) and we want to know if we can play outside. This is Bayes classification with multiple features, as you've recognized.

The image below is a slide from my course at uni, however I don't understand anything of it.

Slide from my course

Who can explain to work out the above formulas to me like I'm five, maybe with Python code? I'd like to understand how I can do naïve bayes classification for multiple features.

  • $\begingroup$ There is no naive Bayes theorem, there are naive Bayes algorithm and Bayes theorem. What exactly is unclear for you? $\endgroup$ – Tim Jan 17 '18 at 17:40
  • $\begingroup$ @Tim It's about the Bayes Theorem with multiple features, as the image shows. I'd like an explanation of the steps I should take to do that and maybe some code :) $\endgroup$ – jbehrens94 Jan 18 '18 at 11:58
  • $\begingroup$ Your image related to naive Bayes algorithm... what exactly is unclear for you in here? Have you seen stats.stackexchange.com/questions/21822/… or stats.stackexchange.com/questions/33185/… ? $\endgroup$ – Tim Jan 18 '18 at 13:16
  • $\begingroup$ @Tim I understand the algorithm for one feature ( P(Class|feature) = P(feature|Class)*P(Class)/P(feature)). However, when I get multiple features (for example outlook, wind and humidity) I get lost with the formula in the image I included. I just don't understand what to do, how to tackle this. $\endgroup$ – jbehrens94 Jan 18 '18 at 13:23
  • $\begingroup$ @Tim I added more detail to my question, do you think you could help me out with the question? $\endgroup$ – jbehrens94 Jan 18 '18 at 14:50

Naive Bayes algorithm assumes that your features are independent (hence we call it "naive", since it makes the naive assumption about independence, so we don't have to care about dependencies between them). What follows, we model

$$ \begin{align} p(C_k, x_1, x_2, ..., x_n) &\propto p(x_1 | C_k) \, p(x_2 | C_k) \dots p(x_n | C_k) \, p(C_k) \\ &= \prod_{i=1}^n p(x_i|C_k) \, p(C_k) \end{align} $$

This follow from the Bayes theorem and independence. So in your example today = (sunny, cool, high, strong), you look at $p($day = sunny $|$ play = yes $)$, and $p($outlook = cool $|$ play = yes $)$, etc.

For more details see the great Wikipedia article on naive Bayes algorithm, the Understanding Naive Bayes thread on our site and the A simple explanation of Naive Bayes Classification thread on StackOverflow.com.

  • $\begingroup$ Let me check if I understand correctly: P(Class) multiplied by P(feature 1|Class) x ... x ... P(feature n|Class) for every feature? $\endgroup$ – jbehrens94 Jan 18 '18 at 15:21
  • $\begingroup$ @jbehrens94 exactly, see the links for details and examples $\endgroup$ – Tim Jan 18 '18 at 15:24
  • $\begingroup$ Thanks, Tim! I've got just one more question, in the image in my question, there's something about estimation, how does that work? Why and when would I do that? $\endgroup$ – jbehrens94 Jan 19 '18 at 8:28
  • $\begingroup$ @jbehrens94 see the links I provided. TLDR: naive Bayes algorithm uses empirical probabilities (observed fractions) and then classifies by choosing the class that has greatest a posteriori probability (that's the argmax part). As simple as that. $\endgroup$ – Tim Jan 19 '18 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.