I am confused with the condition of independence. According to Wiki https://en.wikipedia.org/wiki/Naive_Bayes_classifier, Naive Bayes assumes strong independence between features, so that for a given vector $x = (x_1, x_2, ... ,x_d)$, $d$ is the dimension of the features, we can write $p(x_i|x_{i+1}, ..., x_d, C_k) = p(x_i|C_k)$ since we assumed independence between features, then we have the likelihood \begin{equation*}p(x_1, x_2, ...,x_d|C_k)=\prod_{i=1}^{d} p(x_i|C_k)\end{equation*}however, in this slides page 8: http://www.cs.toronto.edu/~urtasun/courses/CSC411/08_generative.pdf, it assumes data points are independent and identically distributed, so that for a given data set $x = (x_1, x_2, ... ,x_n)$ (each $x_i$ is a vector), we can write \begin{equation*}p(x_1, x_2, ...,x_n|C_k)=\prod_{i=1}^{n} p(x_i|C_k)\end{equation*} So my questions are, why on wiki it assumes independence on feature but on the class slides it assumes $iid$ on data points? I would think assuming independence on data makes more sense, since you want to get a loss function you can take derivative on($log$ the product enables you to do so), I don't know why we might also assume independence on features. And if we do also assume independence on features, does that mean for a multivariate Gaussian Naive Bayes, the covariant matrix must only be diagonal since you can't have relation between features?


1 Answer 1


You are mixing together two "independent" assumptions about independence, and the notation isn't helping - in one case $x_i$ is a feature, in the other $x_i$ is an observation. In the first (Wiki) case, we are assuming features are strongly independent because that allows the probability distribution to separate across features. However, implicit in this is the assumption that the observations are also all independent. If this weren't the case, observation $i$'s features, $x_{ki}$, would in some way "depend on" some other observation $j$'s features, say $x_{lj}$, and you wouldn't be able to write:

$p(x|C) = \Pi_{j=1}^n\Pi_{i=1}^d p(x_{ij}|C_{kj})$

where I'm admittedly being sloppy about $C$-related notation. This relationship is essentially a combination of the two you've written out, and captures the entire cross-observation and cross-feature model that underlies the Naive Bayes classifier.

The slides, on the other hand, make explicit the assumption that the observations are independent, but that particular slide does not capture the assumption of strong independence between features.

  • $\begingroup$ thanks for clarifying this, my last question is that, since features are assumed independent then does it mean for a multivariate Gaussian Naive Bayes, i.e. the likelihood is a multivariate Gaussian, the covariant matrix of this multivariate Gaussian can only be diagonal? (since nonzero off-diagonal entries of $\Sigma$ would imply correlation between features) $\endgroup$
    – Sam
    Sep 23, 2016 at 18:55
  • $\begingroup$ Right, you've got it. $\endgroup$
    – jbowman
    Sep 23, 2016 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.