Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a set of true positive (TP) values which are used to train a model.

I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and 1/5th for testing)

I repeat this using different 1/5ths as the test set. For each run, I have a large set of mixed true positives / true negatives which I use my trained model to attempt to classify. I then obtain an ROC curve. This is done for each run of the cross validation (i.e. I end up with 5 ROC curves)

I then average the AUC and return it.

My problem:

I have two methods of classification: call them method A and method B. for each method, I get 5 ROC curves. How can I determine which method gives me a better ROC if I have more than one ROC for each?

I know computing the AUC and averaging for each method, then comparing averaged AUCs is NOT a good approach.

Note: I have more than 1 model (roughly 120). I just explained in terms of one model for simplicity. So I have 120 models, each one having classified the data using method A and method B, and for each method A & B there are 5 ROCs from cross validation.

Edit


My problem more specifically is that I have >100 sets of sequences, and for each set I construct a position weight matrix, which I then use to score against all sets merged together. I have several scoring schemes so I'd like to determine which ones give me the best classification. For this, I use cross validation: split my data into 5 for each set, train my pwm with 4/5ths of the data and test it on 1/5th. Pool the results from 5 runs, and plot an AUC.

share|improve this question
    
Do you care to tell us how many independent samples (cases) you have? –  cbeleites Nov 15 '12 at 22:32
    
Please explain the relevance of ROC curves in this context. –  Frank Harrell Nov 15 '12 at 22:41

2 Answers 2

There is more to k-fold CV than you do. In essence, the idea of using those crazy splits instead of simply making a few random subsamples is that you can reconstruct the full decision and compare it with original just like you might have done with a predictions on a full train set.

So, sticking to a full k-fold CV mechanism, you just have to merge the predictions from all folds and calculate the ROC for that -- this way you get a single AUROC per model.

However, note that just having two numbers and selecting greater is not a statistically valid way of making comparisons -- without spreads of those two you can't invalidate the hypothesis that both accuracies are roughly the same. So if you are sure you want to do any model selection, you'll need to get those spreads (for instance by bootstrapping the k-fold CV to actually get several AUROC values per classifier) and do some multiple comparison test, probably non-parametric.

share|improve this answer
    
Thank you! I never thought of that way of combining the data. Would it give me the same effect as averaging? I understand its not statistically valid to compare AUC values. I will probably use DeLong's method. However, with so many models, each comparison takes quite a while. –  Omar Nov 16 '12 at 16:48
    
Currently, I'm using the pROC package in R, and the roc.test method. But the wait time is too long –  Omar Nov 16 '12 at 16:54
    
@mbq - I am very interested in your last sentence "probably do some multiple comparison test, probably non-parametric". Would you mind elaborating on that? –  user023472 Jul 17 '13 at 13:57

Just to chime in the @mbq's multiple testing: if you want to compare each of 120 models with each other, that is 7140 comparisons!

You may want to reduce the number of models beforehand by your expert knowledge of the problem. Or include a (few) models that give you baseline performance (constant prediction, random prediction) to test where they fall in the range of all those other models.

Also, make sure you have an independent test set left if you want to report the final performance of the chosen model. Data-driven optimization means that information from the test samples enter your final model as you choose a model that performs well for these (CV) test sets.


Update:

  • Omar, take Frank's hint seriously and read about other performance measures that are better suited.

  • If you decide to stay with AUROC, make sure you calculate them in a range of sensitivities and specificites that are sensible for your application.

  • as mbq says, calculate the spread of your AUROC values of one model, and then think whether you have any chance to identify a good model out of 120 models with 100 independent test cases.

  • In any case: If you want to be able to claim a performance of the final model, you need to test that with a completely independent test set. Samples that were tested for parameter optimization or model selection are not independent any longer. And you should report the uncertainty on this final

share|improve this answer
    
I can't reduce the number of models. Each model has to be tested. Each model classifies different things. More specifically, my models are position weight matrices which represent different proteins. Thus, I need a proxy of how well the PWM picks up its target in a set of possible targets. –  Omar Nov 16 '12 at 16:50
    
@Omar: different things being different input variables, i.e. you want to do a variable selection? Can you explain your application a bit more in the question? –  cbeleites Nov 17 '12 at 10:33
    
I modified the question. Does this make sense? –  Omar Nov 21 '12 at 16:54
    
The relevance of ROC curves for this problem still eludes me, and it also seems there is a problem with the use of improper accuracy scoring rules. –  Frank Harrell Nov 21 '12 at 17:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.