Measuring information content of a random variable in Naive Bayes classifier

Question

I'm trying to improve accuracy in a Naive Bayes classifier that uses a bunch of features. I have a hunch that removing some features may actually improve performance. My reasoning is for a particular feature the estimated PDFs across the classes may be different slightly because of the limited amount of learning data, not because they really are different. For example if a feature accross 2 classes had the same histogram except for some noise

|                      |
|||                  |||    
|||||||_ noise _|||||||| p(x1 | C1), or H1(x)

|                      |
|||                  |||
||||||||________|||||||| p(x1 | C2), or H2(x)

I'm already aware that when modelling p(x1 | C1) I could have chosen a larger bin-width to smooth out the noise, but let's say this was an "optimal" bandwidth in the example above. I want to identify these kinds of cases and remove the feature.

I looked at was Kullback–Leibler divergence. But, I can't see how to compute it because $KL(H2 || H1) = \sum H_2(i) log( \frac{H_2(i)}{H_1(i)}) $ is not defined for example histograms above because of the zeroes in one case where H2(x) is not zero. Or vise versa for KL(H1||H2).

Any suggestions, thoughts? Thanks.

Answer 1

There are different smoothing techniques you can apply to your data to sidestep the 0-observation case you are explaining here.

In fact, this page here describes how to handle your exact case, ie. "probability 0" events when calculating the KL divergence using "absolute discounting".

This PDF enumerates a number of other smoothing methods you might consider in the NLP context, as well.