I am just getting started with trying to understand the theory behind Naive Bayes a bit.

$Y$ = boolean-valued rv $X_i$ = boolean-valued rv (part of random vector $\vec{X}$).

From what I understand, we want this:

$$ P(Y|X_1,X_2,X_3) $$

It can be rewritten as

$$ P(X_1,X_2,X_3|Y)P(Y) $$

The denominator can be ignored.

To get $P(X_1,X_2,X_3|Y)$ We have to do $(2^3-1)*2$ calculations.

And I think I get that. We have to this for every possible combination of values of $X_1,X_2,X_3$ (except one), once for $Y=1$ and once for $Y=0$.

However, if I understand correctly

$$ P(X_1,X_2,X_3|Y) = P(X_1|X_2,X_3,Y)*P(X_2|X_3,Y)*P(X_3|Y) $$

Doesn't this sum up to $2^4 + 2^3 + 2^2 $ calculations?

And then once we assume cond. independence, it looks like this:

$$ P(X_1,X_2,X_3|Y) = P(X_1|Y)*P(X_2|Y)*P(X_3|Y) $$

And we only have to do 2 + 2 + 2 calculations, right?

  • $\begingroup$ Number of calculations to calculate what? You ask about all the conditional probabilities, or a single probability? For single probability, you need just count & divide by total. $\endgroup$ – Tim Nov 21 '20 at 13:14
  • $\begingroup$ I mean the number of parameters you have to calculate with and without conditional independence. What is mentioned here on page 4: cs.cmu.edu/~tom/10601_sp09/lectures/NBayes-1-28-2009-ann.pdf $\endgroup$ – user3629892 Nov 21 '20 at 17:09
  • $\begingroup$ Than $P(X_1,X_2,X_3|Y)$ is a single parameter, while assuming independence you need to calculate three probabilities and multiply them, but the point is not that independence needs less operations, but that calculating the individual probabilities is much easier and needs much less data than for calculating joint probability. $\endgroup$ – Tim Nov 21 '20 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.