# How to be absolutely sure that features do have predictive power to predict the labels (without domain knowledge) ? Does Mutual information help?

Im working on a classification problem which has a severe class imbalance (Its more like an anomaly detection at this point since majority class constitutes 97.5 percent of the dataset ) . Ive tried a couple of approaches to start out but with no luck (ive gotten stuck at one point). So i was beginning to question the features themselves ...

What i did was calculate the mutual information of features with the target class using sklearn's mutual_info_classif and sorted the features in descending order of mutual information . The results seemed wierd to me (or probably not that surprising considering the bad results )as the highest mutual information that i get from any feature is 0.00631345772217 . Does this mean that my data is worthless and i should probably look for more data ?

Does this mean that my data is worthless and i should probably look for more data ?

No, a small mutual information between a target variable and single features does not render your dataset worthless since it neglects the information contained in the combination of features.

I will give a most simple example (XOR problem):

Assume a classfication problem with four datapoints such as:

data = np.array([[1, 2],
[1, 1],
[2, 2],
[2, 1]])


And four associated labels such as:

label_num = [1, 2, 2, 1]


The problem can be visualized like this:

Evaluating the features using mutual information MI(feature, target) yields a mutual information of 0 in both cases.

from sklearn import metrics
metrics.mutual_info_score([1, 1, 2, 2], label_num)


0.0

metrics.mutual_info_score([2, 1, 2, 1], label_num)


0.0

Yet the problem is easy since combining both features allows efficient seperation of both classes such as explained in detail here.

• wow .. what a beautiful example !! , i understand this Xor problem but never thought of it from this perspective . Taking hint from your example would you suggest I continue with feature engineering ? could you suggest some good books (advanced level ) that explicitly talk about feature engineering .. – Vaibhav Arora May 5 '17 at 19:44
• Before you go into depths of feature engineering, I recommend you to try several different models first as a classficiation problem and secondly as an outlier detection problem. Unfortunately I do not have any text book recommendation regarding feature selection and engineering, yet reading papers you can find on google scholar on these topics would be a good next steps – Nikolas Rieble May 7 '17 at 21:24
• Thanks a ton for your replies !! Ive tried everything from SMOTE, to under sampling to every other thing recommended out there. No amount of hyperparameter optimisation is working for me . Ill keep looking and ill post if i find something . – Vaibhav Arora May 8 '17 at 10:07