Generating the ROC curve for ensemble Classifier

I have 3 classifier models namely Logistic Regression, Linear-SVM, Decision Trees as an ensemble technique. I am using majority voting as the classifier combination method for prediction. But when I try to calculate area under ROC curve in R, the function roc needs response and predicted probabilities(score). The predicted class labels can be calculated by majority voting technique but how to calculate the score and thus calculate area under roc curve and generate roc curve?

• Why are you attempting to compute AUC for a classifier? In building an ROC curve on a posterior model, you are generating confusion matrices at varying thresholds on class membership posteriors. For a classifier, the "threshold" is built into the model, and you just get one confusion matrix; a classifier produces less information than a posterior model.
– khol
Apr 24 '18 at 17:12

Yes, you need score-type output to compute a receiver-operating-curve. With fixed cutoff, you just get a single point.

However, with an aggregated model, you can use the individual votes as score. I.e. if you aggregate the votes of $$n$$ classifiers, they form a score ranging from 0 positive to $$n$$ positive votes in the prediction.

For the aggregated prediction of 3 models, the ROC will only have few points corresponding to the at most $$n + 1$$ different predicted scores you observe, but that is still a valid ROC.

This is from the documentation on your roc function:

Description

This is the main function of the pROC package. It builds a ROC curve and returns a “roc” object, a list of class “roc”. This object can be printed, plotted, or passed to the functions auc, ci, smooth.roc and coords. Additionally, two roc objects can be compared with roc.test.

Usage

roc(...)

S3 method for class 'formula':

roc(formula, data, ...)

Default S3 method:

roc(response, predictor, controls, cases, density.controls, density.cases, levels=base::levels(as.factor(response)), percent=FALSE, na.rm=TRUE, direction=c("auto", "<", ">"), algorithm = 1, quiet = TRUE, smooth=FALSE, auc=TRUE, ci=FALSE, plot=FALSE, smooth.method="binormal", ci.method=NULL, density=NULL, ...)

Arguments

response: a factor, numeric or character vector of responses, typically encoded with 0 (controls) and 1 (cases). Only two classes can be used in a ROC curve. If the vector contains more than two unique values, or if their order could be ambiguous, use levels to specify which values must be used as control and case value.

predictor: a numeric vector of the same length than response, containing the predicted value of each observation. An ordered factor is coerced to a numeric.

It looks like you have your predictor vector generated from the majority vote results which leaves the response vector. From the documentation it almost seems like it's a vector denoting which data elements are training and which are testing.

You also have this option:

controls, cases:

instead of response, predictor, the data can be supplied as two numeric or ordered vectors containing the predictor values for control and case observations.

So you can break it up that way as well.

ROC relies on the concept of some adjustable 'threshold', that you can vary, like metaphorically turning a nob, in order to adjust the trade-off between precision and recall.

• precision: choose only examples where you are really certain, will give high precision, but doing this will lead to low ...
• recall: recall means ensuring all positive examples are chosen, at the likely expense of choosing a bunch of examples to be positive, which really are in fact negative, ie 'fast positives'

To trade off between these two extremes, you typically need something that outputs a real number, almost certainly representing a probability, and between zero and one (inclusive). Then you tweak the threshold, to set the probability above which you consider the output to be 'positive', and below which is 'negative'. eg, if output is 0.7, and threshold is 0.5, this would be positive, a 'yes'. But if threshold is 0.8, this would be negative, a 'no'. By adjusting the threshold, you can vary the percentage of samples predicted to be positive of negative. If we assume that the examples which your network assigns a higher probability to are in fact more likely, then adjusting the threshold higher increases the precision, at the expense of recall. Reducing the threshold increases recall, number of true examples judged to be positive, at the expense of precision: ie you'll pick a bunch of examples to be positive that are in reality negative.

To enable such a threshold, you do in fact need real outputs.

One possibility in an ensemble situation would be to average the outputs from each network, rather than using majority voting. Of course, this will change the resulting choice by your network, since the output function is no longer the same. Will enable you to trade-off precision and recall though, and draw a ROC curve.

Note that in addition to requiring a real output, ROC typically is used in a situation where you only have two classes, 'positive' and 'negative'. If you have more than two classes, you might need to create a new 'other' class. Anything in 'other' is 'negative', and anything correctly in one of the other classes is 'positive'.