6
$\begingroup$

Im working on a classification problem which has a severe class imbalance (Its more like an anomaly detection at this point since majority class constitutes 97.5 percent of the dataset ) . Ive tried a couple of approaches to start out but with no luck (ive gotten stuck at one point). So i was beginning to question the features themselves ...

What i did was calculate the mutual information of features with the target class using sklearn's mutual_info_classif and sorted the features in descending order of mutual information . The results seemed wierd to me (or probably not that surprising considering the bad results )as the highest mutual information that i get from any feature is 0.00631345772217 . Does this mean that my data is worthless and i should probably look for more data ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Does this mean that my data is worthless and i should probably look for more data ?

No, a small mutual information between a target variable and single features does not render your dataset worthless since it neglects the information contained in the combination of features.

I will give a most simple example (XOR problem):

Assume a classfication problem with four datapoints such as:

data = np.array([[1, 2],
       [1, 1],
       [2, 2],
       [2, 1]])

And four associated labels such as:

label_num = [1, 2, 2, 1]

The problem can be visualized like this:

enter image description here

Evaluating the features using mutual information MI(feature, target) yields a mutual information of 0 in both cases. 

from sklearn import metrics
metrics.mutual_info_score([1, 1, 2, 2], label_num)

0.0

metrics.mutual_info_score([2, 1, 2, 1], label_num)

0.0

Yet the problem is easy since combining both features allows efficient seperation of both classes such as explained in detail here.

Improve this answer
$\endgroup$
6
  • $\begingroup$ wow .. what a beautiful example !! , i understand this Xor problem but never thought of it from this perspective . Taking hint from your example would you suggest I continue with feature engineering ? could you suggest some good books (advanced level ) that explicitly talk about feature engineering .. $\endgroup$
    – Vaibhav Arora
    Commented May 5, 2017 at 19:44
  • $\begingroup$ Before you go into depths of feature engineering, I recommend you to try several different models first as a classficiation problem and secondly as an outlier detection problem. Unfortunately I do not have any text book recommendation regarding feature selection and engineering, yet reading papers you can find on google scholar on these topics would be a good next steps $\endgroup$
    – Nikolas Rieble
    Commented May 7, 2017 at 21:24
  • $\begingroup$ Thanks a ton for your replies !! Ive tried everything from SMOTE, to under sampling to every other thing recommended out there. No amount of hyperparameter optimisation is working for me . Ill keep looking and ill post if i find something . $\endgroup$
    – Vaibhav Arora
    Commented May 8, 2017 at 10:07
  • $\begingroup$ @Vaibhav Arora: without under- & oversampling - Separating hyperplane for unbalanced classes... I think, TunedThresholdClassifierCV can also be useful for unbalanced ds -- didn't test... $\endgroup$
    – JeeyCi
    Commented Jun 11 at 14:19
  • $\begingroup$ 3rd approach for unbalanced ds, that can be used with SVM.SVC is giving param class_weight='balanced' or sample_weights $\endgroup$
    – JeeyCi
    Commented Jun 11 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.