Naive Bayes model diagnostics -- testing independence between features

Question

One of the main assumptions of the naive Bayes model is that the features are independent. This allows probabilities to be estimated. However, often times it is understood that this assumption doesn't hold in the data. Naive Bayes is, however, still used for applications like text classification because it still provides "good results."

My question is:

1) Is there any way to check how well the assumption holds? I.e., can I test for conditional independence?

2) Is there a way to determine what "good results" are. I.e., are "good results" from a naive Bayes model, vs any other model, based on things like accuracy/how well the model cross validates on training data?

Answer 1

You don’t check the assumption of independence when using Naive Bayes, you don’t even expect it to hold. The textbook example of using Naive Bayes is spam detection with text data, we don’t use it in such cases because we assume that words in human language appear in random combinations, that are independent, but because the algorithm has proven to work in such setting. Moreover, it would be hard if not impossible for you to find data where features are independent, usually there exists some, non-zero degree of dependence.

Answer 2

1) There is no single answer to this question. A good metric for linear dependence is a correlation matrix. i.e. cor(x) in R. If the features are linearly independent, they are uncorrelated, hence the non-diagonal entries of the correlation matrix should be close to zero. See here for a discussion of why naive bayes does well even when its assumptions are violated.

2) Again, there is no single answer to this question. The most common way to compare machine learning models is to see how well they do on held out data and the most common metric is AUC.