5
$\begingroup$

I have a ROC curve generated for a multivariate logistic regression. Does it look correct?

This is what I've done:

  1. Solve $\theta_0 + \theta_1X_1 + \theta_2X_2 ... = Y$ for the $\theta$s
  2. Iterate over all the input $X_i$ and compute the predicted $Y_i'$ for various cutoff values (thresholds) from $0 - 1$, incrementing by $0.01$
  3. For every $X_i$ and the predicted $Y_i'$, compare with original $Y_i$ to get the False Positive (FP), False Negative (FN), True Positive (TP) and True Negative (TN)
  4. Calculate $\text{Sensitivity} = TP/(TP+FN)$ and $\text{Specificity} = TN/(FP+TN)$ for all these values and store them in two different vectors.
  5. Plot the ROC curve for $\text{Sensitivity}$ v/s $\text{Specificity}$ as shown below

Please can someone tell me what is going wrong here?

ROC Curve

$\endgroup$
  • 4
    $\begingroup$ This is definitely not correct. An ROC curve is always increasing. The true positive rate aka sensitivity (y-axis) can never decrease as per your picture. $\endgroup$ – Marc Claesen Aug 8 '13 at 13:44
  • $\begingroup$ Yes, "an ROC curve will always be monotonic (does not decrease in the y direction as we trace along it by increasing x)." More on ROC curves: www0.cs.ucl.ac.uk/staff/ucacbbl/roc $\endgroup$ – Ozgur Ozturk Aug 10 '18 at 14:56
5
$\begingroup$

ROC curve 101

An ROC curve visualizes the predictive performance of a classifier for various levels of conservatism (measured by confidence scores). In simple terms, it illustrates the price you pay in terms of false positive rate to increase the true positive rate. The conservatism is controlled via thresholds on confidence scores to assign the positive and negative label.

The x-axis can be interpreted as a measure of liberalism of the classifier, depicting its false positive rate (1-specificity). The y-axis represents how well it is at detecting positives, depicting the classifier's true positive rate (sensitivity). A perfect classifier's ROC curve passes through $(0,1)$, meaning it can classify all positives correctly without a single false positive. This results in an area under the curve of exactly $1$.

Intuitively, a more conservative classifier (which labels less stuff as positive) has higher precision and lower sensitivity than a more liberal one. When the threshold for positive prediction decreases (e.g. the required positive confidence score decreases), both the false positive rate and sensitivity rise monotonically. This is why an ROC curve always increases monotonically.

Plotting an ROC curve

You need not compute the predictions for various thresholds as you say. Computing an ROC curve is done based on the ranking produced by your classifier (e.g. your logistic regression model).

Use the model to predict every single test point once. You'll get a vector of confidence scores, let's call it $\mathbf{\hat{Y}}$. Using this vector you can produce the full ROC curve (or atleast an estimate thereof). The distinct values in $\mathbf{\hat{Y}}$ are your thresholds. Since you use logistic regression, the confidence scores in $\mathbf{\hat{Y}}$ are probabilities, e.g. in $[0,1]$.

Now, simply iterate over the values and adjust TP/TN/FP/FN as you go and you can compute the ROC curve point by point. The amount of points in your ROC curve is equal to the length of $\mathbf{\hat{Y}}$, assuming there are no ties in prediction.

To plot the final result, use a function that plots in zero order hold (ZOH), rather than linear interpolation between points, like MATLAB's stairs or R's staircase.plot. Also keep this in mind when computing the area under the curve (AUC). If you use linear interpolation instead of ZOH to compute AUC, you actually end up with the area under the convex hull (AUCH).

$\endgroup$
  • 1
    $\begingroup$ Oh, that means I was doing it entirely incorrect! So, as I understand from your reply, the $Y_i'$ I mentioned is your $\mathbf{\hat{Y}}$, which is my predictor's value for the inputs and vary from 0-1 as they are probabilities. Now, I should sort these values and take thresholds, say 0.1, 0.2..0.9 and then calculate all the TP's etc in $Y_i'$. Will try it out quickly and let you know. Much thanks for the reply, very nicely explained $\endgroup$ – sppc42 Aug 8 '13 at 14:05
  • $\begingroup$ @sppc42 yes, that is correct. $\endgroup$ – Marc Claesen Aug 8 '13 at 14:08
  • 1
    $\begingroup$ Somewhat in general, but you must have a specific need in mind if you want to use this high ink:information ratio display. What course of action will its shape lead you to take? $\endgroup$ – Frank Harrell Aug 8 '13 at 19:58
  • 1
    $\begingroup$ @FrankHarrell An ROC curve of a single model is rarely worth its space, but the curves become very useful when comparing several models. AUC is a common summary, but a great deal of information is lost. Two models with identical AUC may work well in entirely different settings when their ROC curves cross. Different settings are common, for example in medical tests, e.g. screening (high recall) vs. diagnostic (high precision). The suitability of classifiers for a given task can be deduced readily from ROC curves. $\endgroup$ – Marc Claesen Aug 8 '13 at 20:21
  • 1
    $\begingroup$ I don't feel that ROC curves allow satisfactory resolution of the problem when the curves cross. This must be handled by specifying utility/cost/loss functions, then the optimum decision comes from the predicted probability and the utility function and the ROC curve is irrelevant. It is the main job of the modeler, in my view, to provide accurate probability (and other) models. $\endgroup$ – Frank Harrell Aug 8 '13 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.