How can I use KL-divergence to weight features?

Question

I have a naive Bayes classifier with two classes (target and non-target) and distributions for a number of features (the same for both classes).

I know that some features contribute more, or less to the overall classification accuracy. I can use KL-divergence to measure the information gain from feature distributions, but how could I use it to weight features?

Say I have features f1, and f2. I have a KL value for the two features and the class.

$KL(f_1, C_1), KL(f_2, C_1)$

$KL(f_1, C_2), KL(f_2, C_2)$

Where do I go from here?

Answer 1

I think what you are looking for is Mutual Information (MI). This will give you the amount of information that each feature contains about the class, thereby giving you an indication of the importance of the feature (i.e. the higher the value of the MI, the more the feature matters in classifying).

To weight the features, you'll need to normalize these values. Also, be aware that $MI(f1; C) + MI(f2; C) >= MI(f1, f2; C)$. That is, the summation of mutual information over individual features with the class will be greater than or equal to the mutual information of the joint feature space and the class. This is because features f1 and f2 could contain overlapping information, thus using the joint feature space will produce less information than the sum of the individuals.

Answer 2

Mutual Information is one approach, but an alternative is Variation of Information. As a quick start, you might want to track down the paper by Meila. Also, Vinh and Bailey provide a broader perspective of different information theoretic methods.