0
$\begingroup$

I am running through a binary classification problem. My target variable is unbalanced the 1s are the 13% of the whole dataset and 0s 87%. Total number of observation 697, number of features 709. Features are time series (financial indeces).

In here Imbalanced data, SMOTE and feature selection I read that feature selection should be applied before using SMOTE but I am kind of unsure if the balancing is really necessary. And also what kind of procedure if SMOTE (oversampling) or RandomUnderSampler (undersampling).

Thanks, Luigi

Improve this question
$\endgroup$
1
  • $\begingroup$ First evaluate some classifiers on your data, and see how far you go, and where the problems are. To see if imbalance is an issue, you can calculate the confusion matrix. $\endgroup$
    – jpmuc
    Commented Dec 30, 2020 at 11:39

1 Answer 1

Reset to default
1
$\begingroup$

SMOTE creates artificial data-points, and in my opinion, it should always be the last option to try. The reason being "creating artificial data-points."

I would follow these steps:

1 - Test some classifiers with the data you have. If the metrics are good enough for your particular goal, you are done.

2 - If the metrics are not good enough, try to engineer new features that capture information regarding the underrepresented class. For instance, once I worked on a time-series classification problem, I realized my under-represented class was quite sensitive to time information. Adding time features such as day of week and hour of day improved my scores a lot, and I did not need to try approaches like SMOTE or under-sampling.

3 - If adding features not enough, try re-weighting loss function, that is, give more weight to examples of underrepresented class when optimizing model parameters (Note that this approach is not possible if you are using a model with a close form solution for your predictions).

4 - If all above fails, then try under or oversampling. Note that this approach always carries the risk of poor out-of-sample results. The reason being you are changing the distribution of your data and making it different than what you will observe out-of-sample

Finally, the best approach is always to get more data, but this is not possible most of the time

Improve this answer
$\endgroup$
5
  • $\begingroup$ thank for your answer! which matrics would you use to evaluate if the selected feature are good enough? Sensitivity, accuracy, precision, specificity, roc_auc, F1 score? all of them? $\endgroup$
    – Luigi87
    Commented Dec 30, 2020 at 13:21
  • 1
    $\begingroup$ Since you have imbalanced data, it is best to evaluate the model's performance using precision, recall, and F1 scores. In case you are making feature selection in an automated way like grid-search, you can optimize the area under the precision and recall curve instead of the area under the receiver operating characteristic curve(roc_auc) $\endgroup$
    – CheeseBurger
    Commented Dec 30, 2020 at 14:11
  • $\begingroup$ yes I tried with logistic classifier and L1 penalty and grid-search was used for finding the best hyperparameter by setting scoring=roc_auc. After I found the hyperparameter I ran again LG with L1 to check the feature importance. So you are saying to use Precision-recall to estimate the hyperparameter rather than roc_auc, right? $\endgroup$
    – Luigi87
    Commented Dec 30, 2020 at 14:22
  • 1
    $\begingroup$ Yes, roc_auc misleads you in case of imbalance, for an example please refer to this link : github.com/guney1/Logistic-Regression-Tutor./blob/master/… There is a section called class imbalance which shows the risk of using roc_auc. $\endgroup$
    – CheeseBurger
    Commented Dec 30, 2020 at 14:25
  • $\begingroup$ thanks for the link, if I set the class_weight to 'balanced' as LogisticRegression(class_weight='balanced'), is the roc_auc still misleading? or with this it is more reliable? $\endgroup$
    – Luigi87
    Commented Jan 7, 2021 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.