I understand that once the logistic regression model has output probabilities, a cutoff value for classifying probabilities of new observations is decided for a model to optimize some metric like sensitivity, specificity, AUC, etc. Then a confusion matrix for that model is constructed and then that metric to optimize is calculated. I know the cutoff value decision is kind of 'trial and error' to play with to get a value you are comfortable with out of the confusion matrix.

My question: what if you are comparing multiple logistic regression models for performance? Do you use the same cutoff value for the predicted probabilities of each model? Or can you use different cutoff values in each model? I guess my concern is if you tailor the cutoff value to each model you can possibly get them all to be very similar in performance, thus making the cutoff value to key variable in performance, and not something like AIC in the model building stage.

EDIT: To clarify my question -> Should model performance between different models be decided in the training set based on AIC and other criteria, and the winning model projected onto the test set to confirm performance? Or do the best models you have found in training need to all be projected onto the test set to compare performance?

  • $\begingroup$ Still brand new and can't upvote yet, but will when I can! $\endgroup$ – MDornbos Jun 30 '16 at 22:27

You have misunderstood logistic regression. Logistic regression is a probability estimator. Cutoffs and improper accuracy scores should play no role in logistic regression analysis and will result in arbitrariness and loss of power/precision.

When you say 'multiple logistic models' you are implying that you don't know how to specify 'the' model or that you are doing problematic variable selection. Please elaborate on why you need to compare multiple models. Note that such comparisons should be based on gold standard methods such as likelihood-based measures.

  • 2
    $\begingroup$ I have read a number of posts and comments by you along the same lines, and I wonder if you have one seminal answer on the site where you present this topic in depth. $\endgroup$ – Antoni Parellada May 12 '16 at 14:11
  • $\begingroup$ @AntoniParellada you could always buy his book which I am sure he is too modest to mention (Regression Modeling Strategies) $\endgroup$ – mdewey May 12 '16 at 14:25
  • $\begingroup$ I will definitely buy Professor Harrell's book at some point. It is a matter of time. $\endgroup$ – Antoni Parellada May 13 '16 at 1:17

From your question I'm assuming that you have trained a few logistic regression models for classification by fitting them on different independent variables of the same training data. Now, you wish to compare the performance of each model. Since you need a cutoff to arrive at a confusion matrix, hence the query on choice of cutoff.

Choosing cutoffs is not straightforward since the optimal cutoff is based on the costs of false positives and false negatives, which are most often asymmetric in real world problems. Hence, it might need to be calibrated according to acceptable tradeoffs decided along with end users; and fine tuned periodically. Usually, due to high class imbalance, a cutoff of 0.5 is not appropriate as the output may be skewed heavily.

If the different models you're comparing are using different independent variables, then cutoffs will usually vary for the same true positive rate since the output ratio of probabilities will span a different range of values for each model.

In general, it is simpler and more effective to compute an ROC curve and compare the AUC (Area under the curve) for each model on the same test set; and select the model with the highest AUC.

  • $\begingroup$ Thanks! Very helpful. I think the best approach is to train different logistic models and use cross-validation to decide between them. If I then use another method like LASSO or Trees, project the best models of each type to the test set. Finally I can use AUC to choose between the different classes of models from the test data. $\endgroup$ – MDornbos May 17 '16 at 0:16
  • $\begingroup$ @MDornbos are you really comparing models with different independent variables? What is the motivation for that? $\endgroup$ – Matthew Drury Jun 30 '16 at 22:50
  • $\begingroup$ Selection using cross-validation will require an outer resampling loop to get an unbiased estimate of model performance. This will require a lot of computation. $\endgroup$ – Frank Harrell Jul 1 '16 at 0:37
  • $\begingroup$ @MatthewDrury I was trying to compare the performance of different logistic regression models, with different independent variables. I wanted to know the best way to do that. I have found it is metrics like AIC, ROC curves, or Cross Validation of those metrics/other metrics - especially ROC when comparing to other model types. $\endgroup$ – MDornbos Jul 18 '16 at 20:07

After more investigation I can answer my own question. When deciding between logistic regression models, we can use cross-validation and choose the metric of ROC to evaluate the area under the curve of varying cutoffs that build the ROC curve. There is also a cv.glm function that cross validate an error metric to choose between different models.

I have found another good way to evaluate models on the test set, which would be constructing an ROC (pROC package) object, and then using coords, which can return a variety of ROC metrics and using best.method = "closest.topleft" to optimize the metrics returned.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.