I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.

I'm a bit confused by the literature where it is mostly just explained as a valid technique to test goodness of fit, and then how to calculate sensitivities, specificities etc. I'm more interested in the workflow of using ROC to test goodness of fit.

Say I have a multiple logistic regression model with one dependent y and 10 independent x1, x2...x10.

Where do I go from here? Do I split the data up and then compare ROC between them or? I'm just a bit confused on the next step.

  • $\begingroup$ You would fit a model and then use the predicted probabilities to calculate sensitivities and specificities that make up the ROC curve. I am not sure what you mean about splitting up the data...some kind of cross validation? $\endgroup$
    – Dave
    Oct 17 '20 at 13:42
  • 1
    $\begingroup$ Instead of using the (area under the) ROC (AUROC) to evaluate model fit, consider a proper scoring rule like the Brier score, the equivalent of mean-square error for class-membership modeling. See this thread among others on this site. Although AUROC can be OK for evaluating a single model, it is not very sensitive for distinguishing reliably among different models. $\endgroup$
    – EdM
    Oct 17 '20 at 14:22

You can use the ROC curve to find what threshold that should be used for classification, depending on how you value Type 1 and Type 2 errors. Generally, you want the area under the curve to be as large as possible to maximize both sensitvity and specificity, that is what is referred to as the AUC value.

Regarding the workflow, I would use AUC as a means to compare different models when the goal is to make a classification. You should not split up the data, if you for example want to compare your full logistic model, with a logistic model with fewer features, you should use both models to predict on the same data.

  • $\begingroup$ But to compare the models, one would need a testing dataset, wouldn't they? That's what I mean by splitting the data. $\endgroup$
    – Paze
    Oct 18 '20 at 12:37
  • $\begingroup$ Well, a testing data set can be helpful, but it isn’t necessary to make use of the ROC curve. $\endgroup$
    – awend
    Oct 18 '20 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.