I'm trying to interpret examples from a probability perspective and my intuition is telling me Logistic Regression should be used for such a purpose despite the score being weaker than the other classifiers.
What I'm after is an accurate ranking order of photos. So if
- photo1: 0.95
- photo2: 0.6
- photo3: 0.8
We could be fairly certain the ranking is photos 1,3,2... while this may seem obvious all the algos have different ranking orderings but I like the results from LR better but I don't want to be irrationally biased but he concept of distance from the average line seems appropriate in this case.
Which model should I use?
facial images projected to a smaller subspace using a CNN.
- 2589 positive
- 4911 negative
Using stratified 5-fold cross-val mean roc auc
- Random Forest (example wtded): 87 (95 w basic tuning),
- Logistic Regression(example wtded): 81
- KNN: 95
- XGB: 94
Inline with that, I don't understand the following quote from the XGBoost docs:
If you care only about the overall performance metric (AUC) of your prediction Balance the positive and negative weights via scale_pos_weight Use AUC for evaluation
If you care about predicting the right probability In such a case, you cannot re-balance the dataset Set parameter max_delta_step to a finite number (say 1) to help convergence
I thought ROC AUC was derived from the probabilities of the examples. Does this mean the individual examples probabilities aren't as valid in a balanced model (cost sensitive learning)?