1
$\begingroup$

Anyone know how to calculate the most informative features where the attributes are normally distributed using Naive Bayes?

My understanding, at least if you have binary attributes, is that you compute max(Pr(feature=1 |classLabel))/min(Pr(feature=1 | classLabel)), for any class label. This will give you the informativeness of feature =1 over two class labels.

But how would you compute the most informative features using a Gaussian Naive Bayes classifier? Any sources would be much appreciated also.

$\endgroup$

2 Answers 2

2
$\begingroup$

You could estimate the mutual information of each feature $i$ with the class label (also known as the expected information gain),

$$I[C, X_i] = H[C] - H[C \mid X_i].$$

The most informative feature is the one which on average produces the least uncertainty in $C$, which is measured by the entropy $H[C \mid X_i]$. We can estimate the entropy by averaging over data points:

$$H[C \mid X_i] \approx -\frac{1}{N} \sum_n \sum_c p(c \mid x_{ni}) \log p(c \mid x_{ni}).$$

In your case, presumably,

$$p(c \mid x_{ni}) = \frac{\mathcal{N}(x_{ni}; \mu_{ci}, \sigma_{ci}^2)}{\sum_{c'} \mathcal{N}(x_{ni}; \mu_{c'i}, \sigma_{c'i}^2)}.$$

Finding the most informative combination of features is a bit trickier. Say you want to find the three most informative features, then you would have to estimate $H[C \mid (X_i, X_j, X_k)]$ for all $\binom{M}{3}$ possible combinations of three features.

For choosing many features, you could try a greedy approach where you first pick the most informative feature $i$. Then you choose the second feature based on $H[C \mid (X_i, X_j)]$, by fixing $i$ and testing all $M$ possible choices for $j$.

$\endgroup$
2
$\begingroup$

I'd suggest using something similar to "Permutation Feature Importance". Assuming you have a metric that you've decided is the best way to determine how accurate your model is, you can calculate a mean decrease in accuracy for each "permuted" feature. For each feature (and only one feature at a time) shuffle the labels and re-train your model and score your test data recording the accuracy. (Could use k-fold cross validation). Repeat the shuffling of labels 10-100x. Perform this for every feature recording the drop in accuracy. Rank the features by their mean decrease in accuracy, the feature with the largest drop in accuracy could be considered the "most informative".

Please note, many Random Forest packages do something similar. https://arxiv.org/abs/1801.01489

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.