0
$\begingroup$

I have a training set of over a 100,000 points that is used to train a Logistic Regression Classifier (logit, since response is binary). The model is testing/fitted on a test set of 20,000 items. The test set is totally independent.

The ROC AUC value for this model is 0.85 which suggests that this is a good model. But I was not convinced. I picked a threshold $0.5$ (i.e., its classified positive if the model response $> 0.5$, negative if model response $< 0.5$).

At this threshold, I get the confusion matrix:

Confusion Matrix and Statistics

          Reference
Prediction     0     1
         0 33307   679
         1     0     0

               Accuracy : 0.98            
                 95% CI : (0.9785, 0.9815)
    No Information Rate : 0.98            
    P-Value [Acc > NIR] : 0.5102          

                  Kappa : 0               
 Mcnemar's Test P-Value : <2e-16          

            Sensitivity : 0.00000         
            Specificity : 1.00000

So my question is, how good is the model if it is unable to predict a 'positive' class at 0.5 threshold?

My guess would be that the threshold of the model for labelling 'positive' is not $0.5$ in this case. Is this intuitive and make sense? Clearly the ROC AUC value is very high, which means that it does have a good TPR rate at lower thresholds.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Why threshold 0.5 should be used? Why do not use estimated probabilities and take a decision at point where you get needed lift or if you have a profit function, at point of profit maximization? $\endgroup$
    – Analyst
    Aug 20, 2015 at 4:15

1 Answer 1

Reset to default
0
$\begingroup$

Threshold 0.5 is arbitrary, as is every threshold. Logistic regression produces probabilities and these can be used to make an decision for classification purposes if that is needed.

For practical purposes in telemarketing campaign I recently analyzed it was decided that cutoff at 0.2 would suffice when at sample level response rate for our offer was at 0.09. At that decile lift was over 2 and we got enough target population for this to be profitable.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is there a statistical technique I can use to determine what the threshold value is? $\endgroup$
    – masfenix
    Aug 20, 2015 at 21:47
  • $\begingroup$ @masfenix no there is no such thing. But perhaps you could use combination of various metrics. I like directly assigning costs and benefits to various misclassifications and correct classifications. $\endgroup$
    – Analyst
    Aug 25, 2015 at 4:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.