I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice.

However the classsifer started predicting all data points belonging to majority class which caused a problem for me. I then decided to use 'class_weight = balanced' of sklearn package which assigns weights to classes in the loss function. Now I do achieve a decent model with ROC AUC of 0.85.

However I have the following questions :-

  1. Do I need to adjust the predicted probabilities since I messed around with distribution by using the class weight parameter?

  2. In my evaluation set I used stratified split. Is this a good choice or should I have balanced dataset in my evaluation set?

  3. Given both class are equally important is ROC AUC a good metric?

  • $\begingroup$ Balancing classes either with SMOTE resampling or weighting in training as you did is dangerous. You have to be certain that the unseen data you will be predicting with that model will be sampled the same way as your training data and will have the same class distribution. If not, then the accuracy you're getting in training will be superficial. $\endgroup$ – Digio Mar 11 '19 at 16:59
  • $\begingroup$ However, If I don't resample or change the weights Logistic Regression model just predicts all as the majority class. To allow for training I believe re-weighting is necessary. To capture your point I believe adjusting the probability thrown out by classifier should be the answer in such case. $\endgroup$ – Axelius Mar 12 '19 at 5:44
  • $\begingroup$ My understanding is that manipulating the class_weight parameter, or using under / oversampling methods will result in biased probabilities when applied to the raw (unweighted / unsampled) data. See "Calibrating Probability with Undersampling for Unbalanced Classification" for some ideas. $\endgroup$ – songololo Jun 25 '19 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.