There's a whole lot of literature about multi-class extensions for ROC.

I have some presentations with illustrations how the calculation works at softclassval's home page (softclassval calculates sensitivities etc. if you have partial class memberships, also for multiple classes - but that is probably an overkill for your problem).

For sensitivity and specificity, the spelled out definitions lead to a very straightforward extension:

• sensitivity: what proportion of truly class $c$ cases are correctly recognized by the model?
• specificity: what proportion of cases truly not belonging to class $c$ are correctly recognized as not coming from class $c$?

If you think about medical diagnostics/epidemiology, the set up is always multinomial from a philosophical point of view: the normal/healthy/control group in fact is rather a "not this disease" group which may contain a whole lot of other diseases. Sometimes classes are mutually exclusive, more often they are not (having, say, a brain tumour does not mean that you cannot have hepatitis nor does it save you from breaking your arm)

• I use package ROCR to plot ROCs, but there are plenty alternatives in R (e.g. pROC - pROC's home page has a comparison of several R packages dealing with ROC generation in R).

update: @Adam

I know this paper:

Landgrebe, T. C. & Paclik, P. The ROC skeleton for multiclass ROC estimation, Pattern Recognition Letters, 31, 949-958 (2010).
DOI: 10.1016/j.patrec.2009.12.037

which deals with independent classes. Basically with $n$ independent classes, you get an $n-1$ dimensional "surface" in $n$ dimensions spanned by the e.g. sensitivity for each class.

I have not seen (nor searched for) anything about ordered levels.

There's a whole lot of literature about multi-class extensions for ROC.

I have some presentations with illustrations how the calculation works at softclassval's home page (softclassval calculates sensitivities etc. if you have partial class memberships, also for multiple classes - but that is probably an overkill for your problem).

For sensitivity and specificity, the spelled out definitions lead to a very straightforward extension:

• sensitivity: what proportion of truly class $c$ cases are correctly recognized by the model?
• specificity: what proportion of cases truly not belonging to class $c$ are correctly recognized as not coming from class $c$?

If you think about medical diagnostics/epidemiology, the set up is always multinomial from a philosophical point of view: the normal/healthy/control group in fact is rather a "not this disease" group which may contain a whole lot of other diseases. Sometimes classes are mutually exclusive, more often they are not (having, say, a brain tumour does not mean that you cannot have hepatitis nor does it save you from breaking your arm)

• I use package ROCR to plot ROCs, but there are plenty alternatives in R (e.g. pROC - pROC's home page has a comparison of several R packages dealing with ROC generation in R).

update: @Adam

I know this paper:

Landgrebe, T. C. & Paclik, P. The ROC skeleton for multiclass ROC estimation, Pattern Recognition Letters, 31, 949-958 (2010).
DOI: 10.1016/j.patrec.2009.12.037

which deals with independent classes. Basically with $n$ independent classes, you get an $n-1$ dimensional "surface" in $n$ dimensions spanned by the e.g. sensitivity for each class.

Here's something about ordered levels:

Nakas, C. T. & Yiannoutsos, C. T. Ordered multiple-class ROC analysis with continuous measurements., Stat Med, 23, 3437-3449 (2004).
DOI: 10.1002/sim.1917

But I cannot access it, so I can't tell you anything further.

There's a whole lot of literature about multi-class extensions for ROC.

I have some presentations with illustrations how the calculation works at softclassval's home page (softclassval calculates sensitivities etc. if you have partial class memberships, also for multiple classes - but that is probably an overkill for your problem).

For sensitivity and specificity, the spelled out definitions lead to a very straightforward extension:

• sensitivity: what proportion of truly class $c$ cases are correctly recognized by the model?
• specificity: what proportion of cases truly not belonging to class $c$ are correctly recognized as not coming from class $c$?

If you think about medical diagnostics/epidemiology, the set up is always multinomial from a philosophical point of view: the normal/healthy/control group in fact is rather a "not this disease" group which may contain a whole lot of other diseases. Sometimes classes are mutually exclusive, more often they are not (having, say, a brain tumour does not mean that you cannot have hepatitis nor does it save you from breaking your arm)

• I use package ROCR to plot ROCs, but there are plenty alternatives in R (e.g. pROC - pROC's home page has a comparison of several R packages dealing with ROC generation in R).

There's a whole lot of literature about multi-class extensions for ROC.

I have some presentations with illustrations how the calculation works at softclassval's home page (softclassval calculates sensitivities etc. if you have partial class memberships, also for multiple classes - but that is probably an overkill for your problem).

For sensitivity and specificity, the spelled out definitions lead to a very straightforward extension:

• sensitivity: what proportion of truly class $c$ cases are correctly recognized by the model?
• specificity: what proportion of cases truly not belonging to class $c$ are correctly recognized as not coming from class $c$?

If you think about medical diagnostics/epidemiology, the set up is always multinomial from a philosophical point of view: the normal/healthy/control group in fact is rather a "not this disease" group which may contain a whole lot of other diseases. Sometimes classes are mutually exclusive, more often they are not (having, say, a brain tumour does not mean that you cannot have hepatitis nor does it save you from breaking your arm)

• I use package ROCR to plot ROCs, but there are plenty alternatives in R (e.g. pROC - pROC's home page has a comparison of several R packages dealing with ROC generation in R).

update: @Adam

I know this paper:

Landgrebe, T. C. & Paclik, P. The ROC skeleton for multiclass ROC estimation, Pattern Recognition Letters, 31, 949-958 (2010).
DOI: 10.1016/j.patrec.2009.12.037

which deals with independent classes. Basically with $n$ independent classes, you get an $n-1$ dimensional "surface" in $n$ dimensions spanned by the e.g. sensitivity for each class.

I have not seen (nor searched for) anything about ordered levels.