I used the functions from this link for creating ROC curve for logistic regression model. Since the object produced by glmer in lme4 package is a S4 object (as far as I know) and the function from the link cannot handle it.

I wonder if there are similar functions for creating ROC curve for multi-level logistic regression model in R.


2 Answers 2


There's a whole lot of literature about multi-class extensions for ROC.

I have some presentations with illustrations how the calculation works at softclassval's home page (softclassval calculates sensitivities etc. if you have partial class memberships, also for multiple classes - but that is probably an overkill for your problem).

For sensitivity and specificity, the spelled out definitions lead to a very straightforward extension:

  • sensitivity: what proportion of truly class $c$ cases are correctly recognized by the model?
  • specificity: what proportion of cases truly not belonging to class $c$ are correctly recognized as not coming from class $c$?

If you think about medical diagnostics/epidemiology, the set up is always multinomial from a philosophical point of view: the normal/healthy/control group in fact is rather a "not this disease" group which may contain a whole lot of other diseases. Sometimes classes are mutually exclusive, more often they are not (having, say, a brain tumour does not mean that you cannot have hepatitis nor does it save you from breaking your arm)

  • I use package ROCR to plot ROCs, but there are plenty alternatives in R (e.g. pROC - pROC's home page has a comparison of several R packages dealing with ROC generation in R).

update: @Adam

I know this paper:

Landgrebe, T. C. & Paclik, P. The ROC skeleton for multiclass ROC estimation, Pattern Recognition Letters, 31, 949-958 (2010).
DOI: 10.1016/j.patrec.2009.12.037

which deals with independent classes. Basically with $n$ independent classes, you get an $n-1$ dimensional "surface" in $n$ dimensions spanned by the e.g. sensitivity for each class.

Here's something about ordered levels:

Nakas, C. T. & Yiannoutsos, C. T. Ordered multiple-class ROC analysis with continuous measurements., Stat Med, 23, 3437-3449 (2004).
DOI: 10.1002/sim.1917

But I cannot access it, so I can't tell you anything further.

  • $\begingroup$ It is easy to make a separate ROC curve for each pair of choices, like 'breast tumour' and 'everything other than breast tumour' Is there any true multilevel ROC curve possible, i.e. one curve for all levels at once? $\endgroup$ Commented Feb 14, 2014 at 22:30
  • $\begingroup$ @AdamRyczkowski: I linked one paper, maybe that gives you a start. $\endgroup$ Commented Feb 15, 2014 at 11:08
  • $\begingroup$ Thank you very much. The articles describe a ROC-surface, with as many dimensions, as there are levels. So technically speaking it is no longer a "curve". $\endgroup$ Commented Feb 15, 2014 at 15:52

Not possible.

The very idea of ROC requires the concept of sensitivity and specificity, which in turn take only real numbers. To have the idea of ROC working with more than two-valued logic, you would need to accept that sensitivity and specificity are vectors.

You might always convert your dependent variable into set two-level dummy variables and perform a series of ROCs. But I guess it's not what you are looking for.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.