1
$\begingroup$

I was going over the derivation of Naive Bayes, and the following 3 lines were given:

Suppose $X = \left < X_1, X_2 \right>$

\begin{align} P(X|Y) &= P(X_1, X_2 | Y) \\[2pt] &= P(X_1 | X_2, Y)P(X_2 | Y) \\[2pt] &= P(X_1 | Y)P(X_2 | Y) \end{align}

So the third line comes from the fact that we have made the assumption that each $X_i$ is conditionally independent given Y, but how was line 2 derived?

$\endgroup$
5
  • $\begingroup$ Would the second line be obvious if you were to erase all instances of "$Y$" from the notation? $\endgroup$
    – whuber
    May 31, 2016 at 1:51
  • $\begingroup$ @whuber Not really. How is the probability of A and B given C equal to the probability of A, given B and C, times the probability of B given C? That can't be Bayes' theorem as the answer suggests. $\endgroup$ May 31, 2016 at 2:15
  • $\begingroup$ If you condition everything on $Y$, then it is not necessary to reference $Y$ explicitly in the notation. The formula reduces to the most basic property of conditional probability. $\endgroup$
    – whuber
    May 31, 2016 at 2:18
  • 1
    $\begingroup$ Oh, so does $P(X_1, X_2)$ mean $P(X_1 and X_2)$? If so, then that's easy! $\endgroup$ May 31, 2016 at 2:21
  • $\begingroup$ I would understand it to mean the joint distribution of $(X_1,X_2)$. $\endgroup$
    – whuber
    May 31, 2016 at 2:23

1 Answer 1

2
$\begingroup$

I've figured it out.

The second line comes from the "chain rule" of probability which states that:

$$P(A, B, \dots, Z) = P(A |B,\dots,Z) \times P(B|C,\dots,Z) \times \dots \times P(Y|Z) \times P(Z)$$

In fact, this can be proven by expanding each term in the equality:

$$P(A, B, ..., Z) = \frac{P(A |B,\dots,Z)}{P(B,\dots,Z)} \times \frac{P(B |C,\dots,Z)}{P(C,\dots,Z)} \times \dots \times \frac{P(Y|Z)}{P(Z)} \times P(Z)$$

and we can see that the terms cancel out.

My main confusion stemmed from the fact that I didn't know that the commas were in fact a notation for the intersection. This post clarifies it:

Now, by common convention, a list of events is interpreted as the intersection (AND) of those events, such that $P(E_1,E_2) = P(E_1 \cap E_2)$ or, using logical connectives instead of set-theory ones, $P(E_1 \land E_2)$. However, that convention is by no means universal, so if you want to be sure to avoid ambiguity, you should explicitly use $\cap$ (or $\land$) to denote the intersection of events.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.