4
$\begingroup$

I am working on classification problem and I have highly imbalanced, but huge data set (I have more than 2mio samples). Now my question is: If I choose subsample of only 15% of the data for training, which is chosen to be balanced, is it OK to use the rest of the samples for testing, even if it is highly imbalanced? Of course I can not look at the accuracy, but some other measure which is taking imbalances into account, e.g. balanced accuracy. I am interested whether the model trained on balanced data is forcing also predictions to "be balanced", i.e. whether the model will force prediction to be balanced. I guess not, but I would like also other oppinions.

models used: logistic regression, decision trees, random forest, MLP with keras

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You can't build proper and reliable model with imbalanced set unless you fix it by undersampling or ovesampling (SMOTE etc.) But you'd better to ask this question in CrossValidated as @schwantke suggested. $\endgroup$
    – maydin
    Jul 29, 2019 at 8:23
  • 1
    $\begingroup$ It doesn't matter on which data you are testing it will give its output, what matters is your data should not be highly bias on any of the class while training.. after training it doesn't matter, only thing to remember is both training and testing data should be scaled and normalize in a same way for training and testing.. $\endgroup$
    – geekzeus
    Jul 29, 2019 at 8:23

1 Answer 1

Reset to default
3
$\begingroup$

In some sense you can use any data that is not part of your test set in any way you like during the training. E.g. balance classes, use data augmentation and so on.

However, your test set should reflect your actual real world use case. So, if in the real world you get very imbalanced classes, you should see how you perform under those conditions.

What you describe seems to fit into the approach, so that sounds fine.

Depending on your prediction model there might be a need to adjust for the true prevalence of each class. Otherwise performance on the test set might suffer from having trained on balanced data. E.g. if you use logistic regression in a two class problem, you would want to adjust the intercept.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you a lot. Do you see also any problems in interpretation of the models (e.g. variable importances in RF or similar) when using balanced data set (artificially forced to be balanced) for training? And how can I adjust the intercept in glm in R ? $\endgroup$
    – pikachu
    Jul 29, 2019 at 11:26
  • 1
    $\begingroup$ Sure, variable importance would change if you did the analysis with or without balancing. Not sure whether that's truly a problem. My initial thought was that adjusting covariates would be easy, e.g. if the minority class is the class you model as "yes, do you not just modify the intercept by adding logit(observed prevalence) - logit(0.5) (where the logit(0.5) is of course 0)? $\endgroup$
    – Björn
    Jul 29, 2019 at 19:45
  • $\begingroup$ Yes, this makes sense, thanks :) I have also found a paper for the LogReg for highly unbalanced data: Gary King and Langche Zeng. “Logistic Regression in Rare Events Data.” Political Analysis 9 (2001): 137-163 , (in case someone reading this post is interested). $\endgroup$
    – pikachu
    Jul 30, 2019 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.