5 votes
Accepted

Binary classification cross validation ROC score - only consider higher confidence class probabilities

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis. It sounds to me ...
5 votes

Binary classification cross validation ROC score - only consider higher confidence class probabilities

The ROC curve and corresponding AUC relate to assessing performance across a spectrum of thresholds, not performance at one particular threshold. Consequently, your plan seems to be equivalent to ...
  • 35.7k
2 votes
Accepted

Predicted Probability with XGBClassifier ranging only from 0.48 to 0.51 for either class

As @Adrian already pointed out, the reason the predicted probabilities are close to 0.5 is because the model has very small number of trees 'n_estimators': 15 and ...
  • 36
1 vote

Could we explain the disadvantage of imbalanced data mathematically?

Not all strongly unbalanced data would result in logistic regression underestimating the probability of the minority. If the data is perfectly separated (and logistic regression is using proper ...
  • 8,021
1 vote
Accepted

What happens if we change the threshold probability value for classifying into different class?

In addition to @Dave's pointer in his comment, I'd like to add simply that a threshold is a hyperparameter. You pick the one that gives you the best compromise between false positives and false ...
1 vote

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

The AUC is the probability that a randomly selected positive is ranked higher than a randomly selected negative. So an AUC of 0.5 is the performance of a classifier that does not rank positives higher ...
  • 81.2k
1 vote

How to *formalize mathematically* that a binary classifier has no predictive performance?

This is complicated, since supervised learning can have so many flavors, but a few general principles can lead you to solve special cases as they arise. The first important topic to consider is what a ...
  • 35.7k

Only top scored, non community-wiki answers of a minimum length are eligible