2
$\begingroup$

I have a heavily imbalanced dataset with 170 columns and 2 million rows, there are also missing data in the set. As practiced, I drop all the null values, normalized the data using min-max method and performed different techniques to address the imbalance. I tried random oversampling, random undersampling, SMOTE, SMOTE-Tomek and SMOTE-ENN, along with this I tried using the data as it is but I placed a class_weight function during training such that class_weight={0:1, 1:1000}.

I trained my fully connected neural networks with an epoch of 30, batch size of 3, binary cross entropy as loss and adams as optimizer.

My cut-off was 60% on specificity and 60% on sensitivity. Here are the results:

                                Sensitivity          Specificty
SMOTE                                8%                   90%
SMOTE-TOMEK                         28%                   65%
SMOTE-ENN                           99%                   0%
Random Oversampling                 23%                   61%
Random Undersampling                20%                   95%
NN with class_weight function       65%                   65%

I would like to ask what is the difference between adding a class_weigh function but using the raw imbalanced data as compared to using the outputs of a re-sampling the imbalanced data during training? What does the class_weight function do? Does it penalizes the weight? if so how?

thanks for the clarifications.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.