# How can I use KL-divergence to weight features?

I have a naive Bayes classifier with two classes (target and non-target) and distributions for a number of features (the same for both classes).

I know that some features contribute more, or less to the overall classification accuracy. I can use KL-divergence to measure the information gain from feature distributions, but how could I use it to weight features?

Say I have features f1, and f2. I have a KL value for the two features and the class.

$KL(f_1, C_1), KL(f_2, C_1)$

$KL(f_1, C_2), KL(f_2, C_2)$

Where do I go from here?

-

To weight the features, you'll need to normalize these values. Also, be aware that $MI(f1; C) + MI(f2; C) >= MI(f1, f2; C)$. That is, the summation of mutual information over individual features with the class will be greater than or equal to the mutual information of the joint feature space and the class. This is because features f1 and f2 could contain overlapping information, thus using the joint feature space will produce less information than the sum of the individuals.