7
$\begingroup$

I knew that, ROC curve are use to assess the performance of classifiers. But is it possible to generate ROC curve for the regression model? If yes, How?

$\endgroup$
  • 5
    $\begingroup$ An ROC curve shows the TPR as a function of FPR. Neither of these measures exists in the context of regression, so there is no such thing as ROC curves for regression. $\endgroup$ – Marc Claesen Mar 5 '14 at 16:46
  • $\begingroup$ While searching for a method of implementing ROC curves for regression I came across the following paper which may be of some use to others wondering the same thing: dl.acm.org/citation.cfm?id=2514279 Hernández-Orallo, José. "ROC curves for regression." Pattern Recognition 46.12 (2013): 3395-3411. $\endgroup$ – Stuart Lacy Apr 11 '15 at 13:34
  • $\begingroup$ Linking a related question: stats.stackexchange.com/questions/105501/… $\endgroup$ – Alexey Grigorev Apr 14 '15 at 8:35
  • $\begingroup$ Late comment: as the answers already pointed, ROC makes no sense here. But its summary statistic, AUC, has an quasi-analogue in regression settings: the Gini coefficient. $\endgroup$ – Firebug Jun 15 '16 at 14:33
  • $\begingroup$ The regression ROC mentioned by Stuart Lacey above is actually implemented in XLMiner. $\endgroup$ – Chris Lloyd Feb 23 '17 at 7:03
11
$\begingroup$

You can't, really.

A (binary) classification task has a small set of possible outcomes: you either correctly detect/reject something or you don't. The ROC curve measures the trade-off between these (specifically, between the false positive rate and the true positive rate). In this setting, there's no notion of "close-but-not-quite-right", but there is often a "knob" you can turn to increase your true positive rate (at the expense of more false positives too), or vice versa.

Regression typically(*) makes continuous predictions. With so many possible outcomes, it's vanishingly unlikely that the model will make an exact prediction (imagine predicting Amazon's annual sales down to the penny--it's not going to happen). There also isn't a TP/FP trade-off.

Instead, people measure a regression model's performance using a loss function, which describes how good/bad a certain amount of error is. For example, a common loss function is the mean-squared error: $\frac{1}{N}\sum_{i=1}^{i=N} (\textrm{obs}_i - \textrm{pred}_i)^2$. This penalizes large errors a lot, but tolerates smaller errors more.


* In some cases, regression can be converted into a classification problem by adding a decision rule. For example, logistic regression, despite the name, is often used as classifier. The "bare" logistic regression output is the probability that an example (i.e., a feature vector) belongs to the positive class: $P(\textrm{class=+} | \textrm{ data})$.

However, you could use a decision rule to assign that example to a class. The obvious decision rule is to assign it to the more likely class: the positive one if the probability is at least a half, and the negative one otherwise. By varying this decision rule (e.g., an example is in the positive class if $P(\textrm{class}=+) > \{0.25, 0.5, 0.75, \textrm{etc}\}$, you can turn the TP/FP knob and generate an ROC curve.

All that said, for most regression tasks, where you're predicting something continuous, ROC analysis is an odd choice.

$\endgroup$
  • 3
    $\begingroup$ Logistic regression is only a classification technique in conjunction with some rule that a predicted probability > $x|x \in [0,1]$ is assigned to one outcome or another. On its own, the logistic regression model is inference on the latent variable Pr(class membership=1). $\endgroup$ – Sycorax Mar 5 '14 at 17:25
  • $\begingroup$ That's a fair point. Interestingly, you could something similar about Naive Bayes (or many other maximum likelihood classifier). How do you feel about the edited version? $\endgroup$ – Matt Krause Mar 5 '14 at 17:58
  • $\begingroup$ I agree with the edit overall. However, I'm not sure what you mean by "you could assign an example to the more likely class if you wanted to perform a classification task instead." What example? Why assign it to a class? I think if you delete that text, it's a great addition to your answer. $\endgroup$ – Sycorax Mar 5 '14 at 18:18
  • $\begingroup$ I'm not sure where I picked this up, but I tend to call one "row" of the data matrix an example. For example, the Fisher's Iris data set has 3 classes (the species of flower), 50 examples per class, and each example has 4 attributes (the length/width of the petal and sepal). $\endgroup$ – Matt Krause Mar 5 '14 at 18:26
  • $\begingroup$ (+1) After the rewrite, I understood -- It makes sense to me now. Regardless, good answer. $\endgroup$ – Sycorax Mar 5 '14 at 18:29
10
$\begingroup$

I haven't enough reputation to make a comment to Matt's comment, that's why I add something via an "answer". Maybe I am wrong, but you can use regression as a classifier, like a logit/probit model, if you have a binary outcome (y variable). Than your "knob", as Matt called it, would be the threshold at which value you choose to see your y* (your continuous prediction of e.g. a linear regression) to be y = 1. Than you can use this threshold for a ROC.

Edit: I agree to the (*) edit of Matt's answer.

Example: There is a continuous variable x and a binary variable y. What you can do is a normal regression of y on x. Then you calculate the predictions of your model dependent on x for each individual, calling these predictions y*. Than you look for a threshold c which does something like $y_{prediction} = \left\{\begin{matrix} 1\text{ if y*} > c \\ 0\text{ else} \end{matrix}\right.$

Than you can use this c for a ROC analysis. (Sorry for my bad formatting, it is my first post here)

$\endgroup$
  • 7
    $\begingroup$ Unless I am missing something, this is basically shoe-horning a classification problem into a regression problem and then evaluating it using classification metrics. $\endgroup$ – Marc Claesen Mar 5 '14 at 17:46
  • $\begingroup$ Agreed; maybe I should make my footnote a little more general. In fact, I think some of the early credit modeling stuff worked like this--linear regression with the output "clamped" to within a certain range, followed by a decision rule. $\endgroup$ – Matt Krause Mar 5 '14 at 17:47
  • $\begingroup$ At least, to my own answer I can do a comment :) @MarcClaesen: I don't say it is the best way to do it. But in our Econometrics class we still had it under "you can do it, with the advantage of using a familiar tool to a new subject; now let's move on to more elaborated stuff for this problem -> logit/probit" $\endgroup$ – user2075339 Mar 5 '14 at 17:57
  • $\begingroup$ By the way, you should have enough reputation to comment everywhere now. Welcome to Cross Validated! $\endgroup$ – Matt Krause Mar 5 '14 at 20:39
2
$\begingroup$

As mentioned by others, you cannot compute the AUC for a linear regression model because there are no true or false positives.

However, AUC is concerned with ranking, more specifically the probability that a randomly-chosen positive sample is ranked higher than a randomly-chosen negative sample. Our predictions are of course numerical and can therefore be ranked, the idea of a positive or negative observation doesn't exist but for a randomly selected pair of predictions there will be a clear positive and negative in that one value will be higher than the other, this is also true in our ground truth values (excluding pairs of identical values). Hence each pair of predictions can be evaluated as being ranked correctly/incorrectly by whether or not they are in the same order as observed in the ground truth.

Repeating this for a sufficient number of predictions and taking the mean number of correct evaluations will provide you a "ranking accuracy" analogous to ROC AUC. A ranking accuracy of 1 means that if you were to order your data points by their predicted value they would be in exactly the same order as if you were to order them by their ground truth value. A ranking accuracy of 0 would mean that the two sequences would be exactly opposite. Important to note is that the actual error on the predicted values could be huge and this metric would not necessarily capture that, for example if all of your predictions were exactly 100x their ground truth values you would observe the maximum ranking accuracy of 1.

This ranking approach to evaluating continuous predictions is used by Rendle et al. in BPR: Bayesian Personalized Ranking from Implicit Feedback (2009) [Arxiv] in deriving their optimization criteria. Section 4.1.1 of that paper demonstrates mathematically the analogies between AUC and this approach.

An interesting metric, useful if the final value is not as important as their relative values (such as in recommendation where the intended output is usually an ordered list).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.