ROC plot for continuous data in R I am currently estimating a bunch of ARMA models, and using them to predict subsets of my data. In order to evaluate their predictive accuracy I would like to make some ROC plots, however since all of my variables are continuous, I wonder how this could be done in R.
Best, Thomas
P.S: I have looked at the ROCR package, but this seems to only work for dichotomous variables.
 A: Well, that is the basis for ROC curves. You see what proportion of correct predictions (i.e. yes or no) are at a variety of predictor levels. The analog of an ROC curve for continuous outcomes would be a validation plot. You develop a prediction score on a training set and validate it on a test set. Or you develop it on the full set and then use bootstrap methods to create neo-samples for validation.
A: The ROC analysis is for binary data, but the AUC of a ROC is just one case of an ordinal rank effect size that can also be used for continuous predictors, which is referred to as Ruscio's A (2008), or the probability of superiority, or several other names. It's a fully non parametric version of the Common Language Effect Size.
I have an implementation of both this statistic and its bootstrapped 95% CIs in an R package here, where it's labelled Ruscio's A effect.
Also see discussion and other implementations here and here.
Finally, if you if you're interested in a full regression framework for ordinal, semi parametric modelling, see the PIM package (De Neve, 2017).
