5
$\begingroup$

I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice.

However the classsifer started predicting all data points belonging to majority class which caused a problem for me. I then decided to use 'class_weight = balanced' of sklearn package which assigns weights to classes in the loss function. Now I do achieve a decent model with ROC AUC of 0.85.

However I have the following questions :-

  1. Do I need to adjust the predicted probabilities since I messed around with distribution by using the class weight parameter?

  2. In my evaluation set I used stratified split. Is this a good choice or should I have balanced dataset in my evaluation set?

  3. Given both class are equally important is ROC AUC a good metric?

$\endgroup$
3
  • $\begingroup$ Balancing classes either with SMOTE resampling or weighting in training as you did is dangerous. You have to be certain that the unseen data you will be predicting with that model will be sampled the same way as your training data and will have the same class distribution. If not, then the accuracy you're getting in training will be superficial. $\endgroup$
    – Digio
    Commented Mar 11, 2019 at 16:59
  • $\begingroup$ However, If I don't resample or change the weights Logistic Regression model just predicts all as the majority class. To allow for training I believe re-weighting is necessary. To capture your point I believe adjusting the probability thrown out by classifier should be the answer in such case. $\endgroup$
    – Axelius
    Commented Mar 12, 2019 at 5:44
  • 1
    $\begingroup$ My understanding is that manipulating the class_weight parameter, or using under / oversampling methods will result in biased probabilities when applied to the raw (unweighted / unsampled) data. See "Calibrating Probability with Undersampling for Unbalanced Classification" for some ideas. $\endgroup$
    – songololo
    Commented Jun 25, 2019 at 21:32

1 Answer 1

1
$\begingroup$

You seem to be mixing up a few notions that often get mixed up in machine learning work. You start out by saying you want the probabilities and use a logistic regression to estimate them. Outstanding! That is exactly what a logistic regression does. Then you say you only predict membership in the minority class.

How did that happen? Logistic regressions do not predict categories. Sure, the predict method in Scikit-learn will predict the category that is most probable, but that is only one of many way to assign categories based on probability predictions. In a binary problem, it might be more appropriate to assign to the minority category if there is a probability of $0.3$ or $0.1$ (maybe even lower) of belonging to the minority category. It's even possible that you would want an "I don't know" zone, so three decisions despite there only being two categories.

Thus, when you write that when you don't apply the weighting you wind up with a logistic regression only predicting the majority category, that is impossible for a model that does not predict categories. It might be that the unweighted logistic regression did a fine job of predicting the probability values. You can check this by assessing the model , such as using a function in Scikit-learn, and you can assess model discrimination (if the model can distinguish between the two categories) using ROCAUC. You say you got a good ROCAUC of $0.85$ after you did the weighted regression. If you did not calculate the ROCAUC for the outputs of the unweighted regression, I suggest doing so and speculate that you will get a value around $0.85$, same as with the weighted loss function.

To address your explicit inquiries...

Do I need to adjust the predicted probabilities since I messed around with distribution by using the class weight parameter?

If you care about the probability predictions being accurate (and there are good reasons to), you should check their accuracy whether you use class weights or not. If you find that there are issues with the probability predictions (a lack of ), there are techniques to address such an issue, such as isotonic regression and Platt scaling.

With class weighting in particular, a simpler calibration approach may be possible following the logic here. The difference is that, instead of giving more weight to the minority class by artificially inflating their count, you gave more weight to the minority class by weighting the loss function. There should be a way to wrangle the linked equations to work for a weighted loss function.

In my evaluation set I used stratified split. Is this a good choice or should I have balanced dataset in my evaluation set?

Yes, your stratified sampling seems reasonable.

The evaluation set should represent the reality of operational conditions (at least as best as you can know them), as that is supposed to be how you get a sense of how good or bad your predictions will be when it counts. If you have imbalance, then you want to make sure you can perform well under the imbalance, not under artificial conditions. Consequently, if you develop an evaluation set that has nine dogs for every cat because the reality is that dogs outnumber cats $9$:$1$, that is a good strategy.

Given both class are equally important is ROC AUC a good metric?

ROCAUC is a measure of how well the categories are distinguished from each other, but ROCAUC is completely indifferent to prediction calibration. In fact, you can divide the predictions by two or two-trillion without changing the ROC curve or the area under the ROC curve. Since you seem to care about the probability predictions, you risk missing problems by considering ROCAUC instead of a metric like Brier score or log loss that also accounts for calibration. These can be normalized as Efron's or McFadden's, respectively, pseudo $R^2$ to allow for an easier interpretation, a notion totally aligned with the $R^*$ "universal coefficient of determination" in equation $32$ of Gneiting & Resin (2023).

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.