# Measuring information content of a random variable in Naive Bayes classifier

I'm trying to improve accuracy in a Naive Bayes classifier that uses a bunch of features. I have a hunch that removing some features may actually improve performance. My reasoning is for a particular feature the estimated PDFs across the classes may be different slightly because of the limited amount of learning data, not because they really are different. For example if a feature accross 2 classes had the same histogram except for some noise

|                      |
|||                  |||
|||||||_ noise _|||||||| p(x1 | C1), or H1(x)

|                      |
|||                  |||
||||||||________|||||||| p(x1 | C2), or H2(x)


I'm already aware that when modelling p(x1 | C1) I could have chosen a larger bin-width to smooth out the noise, but let's say this was an "optimal" bandwidth in the example above. I want to identify these kinds of cases and remove the feature.

I looked at was Kullback–Leibler divergence. But, I can't see how to compute it because $KL(H2 || H1) = \sum H_2(i) log( \frac{H_2(i)}{H_1(i)})$ is not defined for example histograms above because of the zeroes in one case where H2(x) is not zero. Or vise versa for KL(H1||H2).

Any suggestions, thoughts? Thanks.

-