Following the discussion on here I started worrying less about class imbalance. However, I recently started building a predictor, using XGBoost, and I wanted to used LogLoss as my target metric. I have only two classes, with a small imbalance (90 - 10 )
I did nothing to fix that imbalance on my training set. Interestingly, independently of the metric I used (MCC, LogLoss, Lift), not addressing the class imbalance problem always outperform all the other strategies (oversampling, undersampling, SMOTE...).
However, I now face a question. When evaluating the model on the test set, I can compute a LogLoss or a weighted LogLoss (that balances the two classes). I found out that, using a weighted LogLoss tend to drive all the other metrics down...
Digging deeper, I realize that, for models with lower weighted LogLoss, what was happening was that all the probabilities were close to 0.5. For models with lower LogLoss, probabilities were all over the place.
Is there a reason for this? What would be your advice?