PREFACE: I don't care about the merits of using a cutoff or not, or how one should choose a cutoff. My question is purely mathematical and due to curiosity.
Logistic regression models the posterior conditional probability of class A versus class B and it fits a hyperplane where the posterior conditional probabilities are equal. So in theory, I understood that a 0.5 classification point will minimize total errors regardless of set balance, since it models the posterior probability (assuming you consistently encounter the same class ratio).
In my real life example, I obtain very poor accuracy using P > 0.5 as my classifying cutoff (about 51% accuracy). However, when I looked at the AUC it is above 0.99. So I looked at some different cutoff values and found that P > 0.6 gave me 98% accuracy (90% for the smaller class and 99% for the bigger class) - only 2% of cases misclassified.
The classes are heavily unbalanced (1:9) and it is a high-dimensional problem. However, I allocated the classes equally to each cross-validation set so that there should not be a difference between the balance of classes between model fit and then prediction. I also tried using the same data from the model fit and in predictions and the same issue occurred.
I'm interested in the reason why 0.5 would not minimize errors, I thought this would be by design if the model is being fit by minimizing cross-entropy loss.
Does anyone have any feedback as to why this happens? Is it due to adding penalization, can someone explain what is happening if so?