3
$\begingroup$

Using the example at 3Blue1Brown I constructed a table to help me remember Bayes theorem

Librarians and Farmers

where L=Librarian and S =Shy. I understand that $$P(S,L) = P(S|L)P(L) = P(L|S)P(S) = \frac{4}{210}$$ I am trying to use that knowledge to understand the following Naive Bayes Classifier.

Fruit Long Sweet Yellow

The objective is to predict the fruit when only 3 features (long, sweet, yellow) are known. Thus we need the ratios of $$P(B|L,S,Y) : P(O_r|L,S,Y) : P(O_t|L,S,Y)$$ where L=Long, S=Sweet, Y=Yellow, B=Banana, $O_r$=Orange , $O_t$ = Other

For the first value, I understand that $P(B|L,S,Y) = P(B,L,S,Y) \div P(L,S,Y)$

and I am told that $$P(B,L,S,Y)=P(L|B) P(S|B) P(Y|B) P(B) $$

But I do not understand why this is so. Thus I tend to forget the formula when I come to use it.

[Update]

In the Librarian/Farmer example I could construct a table as follows;
IsLibrarian boolean
IsShy boolean
I can read each of the four possible combinations from the table. For example in sql the top left cell would be

select count(*) from myTable where IsLibrarian and IsShy

In the Fruit example I could construct a table as follows;
IsLong boolean
IsSweet boolean
IsYellow boolean
Type $\in \{Banana,Orange,Other\}$

However I cannot read from the table the equivalent of

select count(*) from myTable where IsLong and IsSweet and IsYellow and Type = Banana

From the bottom row I understand that P(L) = .5, P(S) = .65 and P(Y) = .35 So I guess that with the independence assumption then P(L,S,Y) = .5 .65 .35 = .11375

From the right column I see P(B)= .5

[Update after thinking about Tim's answer]

I think it might help me to read P(L|B) as "Long Bananas out of all Bananas" so $$P(B,L,S,Y)=P(L|B) P(S|B) P(Y|B) P(B) $$ becomes
"Long Bananas out of all Bananas" = $\frac{400}{500}$
"Sweet Bananas out of all Bananas" = $\frac{350}{500}$
"Yellow Bananas out of all Bananas" = $\frac{450}{500}$
"Bananas out of all Fruit" =$\frac{500}{1000}$
$=0.252$

Then
"Long Oranges out of all Oranges " = 0 so zero
And

so $$P(O,L,S,Y)=P(L|O) P(S|O) P(Y|O) P(O) $$ becomes
"Long Other out of all Other" = $\frac{100}{200}$
"Sweet Other out of all Other" = $\frac{150}{200}$
"Yellow Other out of all Other" = $\frac{50}{200}$
"Other fruit out of all Fruit" =$\frac{200}{1000}$
$=0.01575$

So the ratios are .252:0:.01575 or $\approx .93:0:.07$

$\endgroup$

1 Answer 1

1
$\begingroup$

This is a toy example, so you could calculate $P(B | L, S, Y)$, $P(B, L, S, Y)$, and $P(L, S, Y)$ from the table. With naive Bayes algorithm, you approximate the conditional probability by assuming conditional independence:

$$ P(L, S, Y | B) \approx P(L | B) \, P(S | B) \, P(Y | B) $$

The approximation may be useful when you cannot calculate the conditional probability directly from the data. This can happen if you don't have data on all the combinations of all the levels of all the variables, or you have insufficient data to calculate them precisely. For example, imagine building a spam detector using the content of e-mails, for non-naive approach, you would need to have data on all the possible combinations of words that can make sentences--that's an infinite amount of data! In such a case, you could instead calculate the individual conditional probabilities and use them with naive Bayes.

$\endgroup$
2
  • $\begingroup$ Even if you could calculate the conditional probability directly from the data the approximation is likely to give you a better estimate due to the individual probabilities having a larger sample than the conditional one $\endgroup$
    – astel
    Commented Oct 19, 2021 at 10:44
  • $\begingroup$ @astel correct. This is mentioned in the linked thread, so I didn't want to repeat myself, but maybe I'll edit for clarity. $\endgroup$
    – Tim
    Commented Oct 19, 2021 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.