I have a training set of over a 100,000 points that is used to train a Logistic Regression Classifier (logit, since response is binary). The model is testing/fitted on a test set of 20,000 items. The test set is totally independent.
The ROC AUC value for this model is 0.85 which suggests that this is a good model. But I was not convinced. I picked a threshold $0.5$ (i.e., its classified positive if the model response $> 0.5$, negative if model response $< 0.5$).
At this threshold, I get the confusion matrix:
Confusion Matrix and Statistics Reference Prediction 0 1 0 33307 679 1 0 0 Accuracy : 0.98 95% CI : (0.9785, 0.9815) No Information Rate : 0.98 P-Value [Acc > NIR] : 0.5102 Kappa : 0 Mcnemar's Test P-Value : <2e-16 Sensitivity : 0.00000 Specificity : 1.00000
So my question is, how good is the model if it is unable to predict a 'positive' class at 0.5 threshold?
My guess would be that the threshold of the model for labelling 'positive' is not $0.5$ in this case. Is this intuitive and make sense? Clearly the ROC AUC value is very high, which means that it does have a good TPR rate at lower thresholds.