I have a heavily imbalanced dataset with 170 columns and 2 million rows, there are also missing data in the set. As practiced, I drop all the null values, normalized the data using min-max method and performed different techniques to address the imbalance. I tried random oversampling, random undersampling, SMOTE, SMOTE-Tomek and SMOTE-ENN, along with this I tried using the data as it is but I placed a class_weight function during training such that class_weight={0:1, 1:1000}.

I trained my fully connected neural networks with an epoch of 30, batch size of 3, binary cross entropy as loss and adams as optimizer.

My cut-off was 60% on specificity and 60% on sensitivity. Here are the results:

                                Sensitivity          Specificty
SMOTE                                8%                   90%
SMOTE-TOMEK                         28%                   65%
SMOTE-ENN                           99%                   0%
Random Oversampling                 23%                   61%
Random Undersampling                20%                   95%
NN with class_weight function       65%                   65%

I would like to ask what is the difference between adding a class_weigh function but using the raw imbalanced data as compared to using the outputs of a re-sampling the imbalanced data during training? What does the class_weight function do? Does it penalizes the weight? if so how?

thanks for the clarifications.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.