Logistic Regression Cutoff Values for Multiple Models

Question

I understand that once the logistic regression model has output probabilities, a cutoff value for classifying probabilities of new observations is decided for a model to optimize some metric like sensitivity, specificity, AUC, etc. Then a confusion matrix for that model is constructed and then that metric to optimize is calculated. I know the cutoff value decision is kind of 'trial and error' to play with to get a value you are comfortable with out of the confusion matrix.

My question: what if you are comparing multiple logistic regression models for performance? Do you use the same cutoff value for the predicted probabilities of each model? Or can you use different cutoff values in each model? I guess my concern is if you tailor the cutoff value to each model you can possibly get them all to be very similar in performance, thus making the cutoff value to key variable in performance, and not something like AIC in the model building stage.

EDIT: To clarify my question -> Should model performance between different models be decided in the training set based on AIC and other criteria, and the winning model projected onto the test set to confirm performance? Or do the best models you have found in training need to all be projected onto the test set to compare performance?

Answer 1

You have misunderstood logistic regression. Logistic regression is a probability estimator. Cutoffs and improper accuracy scores should play no role in logistic regression analysis and will result in arbitrariness and loss of power/precision.

When you say 'multiple logistic models' you are implying that you don't know how to specify 'the' model or that you are doing problematic variable selection. Please elaborate on why you need to compare multiple models. Note that such comparisons should be based on gold standard methods such as likelihood-based measures.

Answer 2

From your question I'm assuming that you have trained a few logistic regression models for classification by fitting them on different independent variables of the same training data. Now, you wish to compare the performance of each model. Since you need a cutoff to arrive at a confusion matrix, hence the query on choice of cutoff.

Choosing cutoffs is not straightforward since the optimal cutoff is based on the costs of false positives and false negatives, which are most often asymmetric in real world problems. Hence, it might need to be calibrated according to acceptable tradeoffs decided along with end users; and fine tuned periodically. Usually, due to high class imbalance, a cutoff of 0.5 is not appropriate as the output may be skewed heavily.

If the different models you're comparing are using different independent variables, then cutoffs will usually vary for the same true positive rate since the output ratio of probabilities will span a different range of values for each model.

In general, it is simpler and more effective to compute an ROC curve and compare the AUC (Area under the curve) for each model on the same test set; and select the model with the highest AUC.

Answer 3

After more investigation I can answer my own question. When deciding between logistic regression models, we can use cross-validation and choose the metric of ROC to evaluate the area under the curve of varying cutoffs that build the ROC curve. There is also a cv.glm function that cross validate an error metric to choose between different models.

I have found another good way to evaluate models on the test set, which would be constructing an ROC (pROC package) object, and then using coords, which can return a variety of ROC metrics and using best.method = "closest.topleft" to optimize the metrics returned.