0
$\begingroup$

I'm trying to understand roc function of pROC package. My first understanding is I need to have a model first and then run against testing set to get predicted probability for each sample, then by feeding with the true class of testing set and probability from the model, with different cutoff we can get the ROC curve, like:

roc <- roc(response = test$Class, predictor = test$ModelProb). 

However looks like with a dataset we can feed roc function directly with predictor and outcome. Say I have a dataset X with predictors numeric1, categorical1 and OutcomeBinaryClass, I can use:

roc(OutcomeBinaryClass, numeric1)

or

roc(OutcomeBinaryClass, categorical1)

or use filterVarImp(X, OutcomeBinaryClass), which is actually a wrapper and apply auc with each predictor of X and Y.

How ROC curve is generated behind the scene without a model (to get the probabilities)? And what is the meaning to get ROC for each predictor?

$\endgroup$
2
$\begingroup$

My first understanding is I need to have a model first

Your understanding is incorrect. You only need any binary classifier on which you can vary the decision threshold. Building a model is a common way to achieve that, but your numeric and categorical variables are valid classifiers as well.

I always recommend the following introductory paper to get started with ROC analysis: Fawcett, T. An introduction to ROC analysis. Pattern Recognition Letters 27, 861–874 (2006).

$\endgroup$
  • $\begingroup$ Can you pls elaborate numeric and categorical variables are valid classifiers as well? How to tell the predicted value and true value with just predictor and outcome? If I have a predictor with values between 0-1, how roc function tell it is just predictor values instead of predicted probabilities? $\endgroup$ – Brad Jan 5 '17 at 5:52
  • $\begingroup$ I rephrased my answer a bit, but ultimately you vary your decision threshold between -∞ and +∞ so exactly where your values lie or whether they are probabilities doesn't matter, only their ordering. $\endgroup$ – Calimo Jan 5 '17 at 8:41
  • $\begingroup$ Sorry still not get it. Can anybody describe a simple logic flow as how to calculate ROC with roc function in pROC package when using roc(SomeSinglePredictor, Outcome)? Thanks. $\endgroup$ – Brad Jan 6 '17 at 9:01
  • $\begingroup$ @Brad you reversed the predictor and response, so it is roc(Outcome, SomeSinglePredictor). This is enough to calculate the ROC curve. $\endgroup$ – Calimo Feb 16 '17 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.