2
$\begingroup$

I'm interested in using ROC to test for goodness of fit for binary models such as logistic regression.

I'm a bit confused by the literature where it is mostly just explained as a valid technique to test goodness of fit, and then how to calculate sensitivities, specificities etc. I'm more interested in the workflow of using ROC to test goodness of fit.

Say I have a multiple logistic regression model with one dependent y and 10 independent x1, x2...x10.

Where do I go from here? Do I split the data up and then compare ROC between them or? I'm just a bit confused on the next step.

Improve this question
$\endgroup$
2
  • $\begingroup$ You would fit a model and then use the predicted probabilities to calculate sensitivities and specificities that make up the ROC curve. I am not sure what you mean about splitting up the data...some kind of cross validation? $\endgroup$
    – Dave
    Commented Oct 17, 2020 at 13:42
  • 1
    $\begingroup$ Instead of using the (area under the) ROC (AUROC) to evaluate model fit, consider a proper scoring rule like the Brier score, the equivalent of mean-square error for class-membership modeling. See this thread among others on this site. Although AUROC can be OK for evaluating a single model, it is not very sensitive for distinguishing reliably among different models. $\endgroup$
    – EdM
    Commented Oct 17, 2020 at 14:22

1 Answer 1

Reset to default
2
$\begingroup$

You can use the ROC curve to find what threshold that should be used for classification, depending on how you value Type 1 and Type 2 errors. Generally, you want the area under the curve to be as large as possible to maximize both sensitvity and specificity, that is what is referred to as the AUC value.

Regarding the workflow, I would use AUC as a means to compare different models when the goal is to make a classification. You should not split up the data, if you for example want to compare your full logistic model, with a logistic model with fewer features, you should use both models to predict on the same data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But to compare the models, one would need a testing dataset, wouldn't they? That's what I mean by splitting the data. $\endgroup$
    – Paze
    Commented Oct 18, 2020 at 12:37
  • $\begingroup$ Well, a testing data set can be helpful, but it isn’t necessary to make use of the ROC curve. $\endgroup$
    – awend
    Commented Oct 18, 2020 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.