Fastest way to compare ROC curves

Question

I have a set of true positive (TP) values which are used to train a model.

I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and 1/5th for testing)

I repeat this using different 1/5ths as the test set. For each run, I have a large set of mixed true positives / true negatives which I use my trained model to attempt to classify. I then obtain an ROC curve. This is done for each run of the cross validation (i.e. I end up with 5 ROC curves)

I then average the AUC and return it.

My problem:

I have two methods of classification: call them method A and method B. for each method, I get 5 ROC curves. How can I determine which method gives me a better ROC if I have more than one ROC for each?

I know computing the AUC and averaging for each method, then comparing averaged AUCs is NOT a good approach.

Note: I have more than 1 model (roughly 120). I just explained in terms of one model for simplicity. So I have 120 models, each one having classified the data using method A and method B, and for each method A & B there are 5 ROCs from cross validation.

Edit

---

My problem more specifically is that I have >100 sets of sequences, and for each set I construct a position weight matrix, which I then use to score against all sets merged together. I have several scoring schemes so I'd like to determine which ones give me the best classification. For this, I use cross validation: split my data into 5 for each set, train my pwm with 4/5ths of the data and test it on 1/5th. Pool the results from 5 runs, and plot an AUC.

Answer 1

There is more to k-fold CV than you do. In essence, the idea of using those crazy splits instead of simply making a few random subsamples is that you can reconstruct the full decision and compare it with original just like you might have done with a predictions on a full train set.

So, sticking to a full k-fold CV mechanism, you just have to merge the predictions from all folds and calculate the ROC for that -- this way you get a single AUROC per model.

However, note that just having two numbers and selecting greater is not a statistically valid way of making comparisons -- without spreads of those two you can't invalidate the hypothesis that both accuracies are roughly the same. So if you are sure you want to do any model selection, you'll need to get those spreads (for instance by bootstrapping the k-fold CV to actually get several AUROC values per classifier) and do some multiple comparison test, probably non-parametric.

Answer 2

Just to chime in the @mbq's multiple testing: if you want to compare each of 120 models with each other, that is 7140 comparisons!

You may want to reduce the number of models beforehand by your expert knowledge of the problem. Or include a (few) models that give you baseline performance (constant prediction, random prediction) to test where they fall in the range of all those other models.

Also, make sure you have an independent test set left if you want to report the final performance of the chosen model. Data-driven optimization means that information from the test samples enter your final model as you choose a model that performs well for these (CV) test sets.

---

Update:

• Omar, take Frank's hint seriously and read about other performance measures that are better suited.

• If you decide to stay with AUROC, make sure you calculate them in a range of sensitivities and specificites that are sensible for your application.

• as mbq says, calculate the spread of your AUROC values of one model, and then think whether you have any chance to identify a good model out of 120 models with 100 independent test cases.

• In any case: If you want to be able to claim a performance of the final model, you need to test that with a completely independent test set. Samples that were tested for parameter optimization or model selection are not independent any longer. And you should report the uncertainty on this final