12
$\begingroup$

I applied logistic regression to my data on SAS and here are the ROC curve and classification table.

enter image description here

I am comfortable with the figures in the classification table, but not exactly sure what the roc curve and the area under it show. Any explanation would be greatly appreciated.

$\endgroup$
18
$\begingroup$

When you do logistic regression, you are given two classes coded as $1$ and $0$. Now, you compute probabilities that given some explanatory varialbes an individual belongs to the class coded as $1$. If you now choose a probability threshold and classify all individuals with a probability greater than this threshold as class $1$ and below as $0$, you will in the most cases make some errors because usually two groups cannot be discriminated perfectly. For this threshold you can now compute your errors and the so-called sensitivity and specificity. If you do this for many thresholds, you can construct a ROC curve by plotting sensitivity against 1-Specificity for many possible thresholds. The area under the curve comes in play if you want to compare different methods that try to discriminate between two classes, e. g. discriminant analysis or a probit model. You can construct the ROC curve for all these models and the one with the highest area under the curve can be seen as the best model.

If you need to get a deeper understanding, you can also read the answer of a different question regarding ROC curves by clicking here.

$\endgroup$
  • $\begingroup$ How is the area under the ROC curve different from the correct rate in the classification table? $\endgroup$ – Günal Nov 30 '14 at 19:49
  • 2
    $\begingroup$ The table shows only the correct and non correct for one threshold. However, the AUROC curve is a measure of the complete classification method and the correct and non correct for many different thresholds. $\endgroup$ – random_guy Nov 30 '14 at 19:57
  • $\begingroup$ Nice to hear that! $\endgroup$ – random_guy Nov 30 '14 at 20:03
6
$\begingroup$

The logistic regression model is a direct probability estimation method. Classification should play no role in its use. Any classification not based on assessing utilities (loss/cost function) on individual subjects is inappropriate except in very special emergencies. The ROC curve is not helpful here; neither are sensitivity or specificity which, like overall classification accuracy, are improper accuracy scoring rules that are optimized by a bogus model not fitted by maximum likelihood estimation.

Note that you achieve high predictive discrimination (high $c$-index (ROC area)) by overfitting the data. You need perhaps at least $15p$ observations in the least frequent category of $Y$, where $p$ is the number of candidate predictors being considered, in order to obtain a model that is not significantly overfitted [i.e., a model that is likely to work on new data about as well as it worked on the training data]. You need at least 96 observations just to estimate the intercept such that the predicted risk has a margin of error $\leq 0.05$ with 0.95 confidence.

$\endgroup$
  • $\begingroup$ @Frank Harrell: Could you elaborate on the calculation regarding the intercept as well as the comment regarding the margin of error. Thanks! $\endgroup$ – julieth Dec 2 '14 at 16:06
  • $\begingroup$ @FrankHarrell does your advice that we need at least 15p observations apply if we end up doing ridge regression to calibrate the model? My understanding is that we replace p then by the effective dimensionality. $\endgroup$ – Lepidopterist Jun 5 '17 at 19:19
  • $\begingroup$ Correct, and I'd say that you use penalization such as quadratic (ridge) penalty to estimate the parameters, which results in better calibration $\endgroup$ – Frank Harrell Jun 7 '17 at 12:24
5
$\begingroup$

The AUC is basically just telling you how frequently a random draw from your predicted response probabilities on your 1-labeled data will be greater than a random draw from your predicted response probabilities on your 0-labeled data.

$\endgroup$
3
$\begingroup$

I'm not the author of this blog and I found this blog extremely helpful: http://fouryears.eu/2011/10/12/roc-area-under-the-curve-explained

Applying this explanation to your data, the average positive example has about 10% of negative examples scored higher than it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.