Naive Bayes Binary Classification with Binary Features

Question

I have a dataset with two classes $C_0$ and $C_1$. I have around $10$ to $20$ features that take binary values (either $0$ or $1$). My dataset has around $10000$ instances, with only a hundred of these belonging to class $C_0$ and all the others to class $C_1$.

I would like to apply some dimensionality reduction technique, actually not to get rid of dimensions but to assess how relevant (distinguishing) are for my problem. I considered applying PCA, but I've read a lot about the topic and seems the community is divided on whether it is legitimate or not to apply PCA on binary data.

What would your recommend in this case?

Answer 1

This problem could presumably be solved many different ways. Side-stepping the naive Bayes component of the question for a moment, it seems to me you need to do "feature selection" rather than dimensionality reduction, as you seem interested in knowing how relevant each of your features are. If you want to assess how important features are for your task, I think you should do that in a supervised setting rather than an unsupervised setting (PCA is unsupervised).

A super-simple approach would be to use logistic regression with an L1 penalty on the weights (or coefficients). Used in this setting, you could do feature selection this way. The L1 penalty on the weights would force some of the coefficients to be zero and only the relevant features would end up with non-zero coefficients. Perhaps this logistic regression approach would yield a good classifier, or you could take the features that had non-zero weights here and use them in another approach (such as naive bayes).

At this point, many of the standard software tools seem to let you use a version of the "lasso" function with binary output data (for matlab, see lassoglm). If you want proper background on this approach, there are some references here (the 2008 paper basically covers the logistic regression setting).