5
$\begingroup$

I have a set of true positive (TP) values which are used to train a model.

I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and 1/5th for testing)

I repeat this using different 1/5ths as the test set. For each run, I have a large set of mixed true positives / true negatives which I use my trained model to attempt to classify. I then obtain an ROC curve. This is done for each run of the cross validation (i.e. I end up with 5 ROC curves)

I then average the AUC and return it.

My problem:

I have two methods of classification: call them method A and method B. for each method, I get 5 ROC curves. How can I determine which method gives me a better ROC if I have more than one ROC for each?

I know computing the AUC and averaging for each method, then comparing averaged AUCs is NOT a good approach.

Note: I have more than 1 model (roughly 120). I just explained in terms of one model for simplicity. So I have 120 models, each one having classified the data using method A and method B, and for each method A & B there are 5 ROCs from cross validation.

Edit


My problem more specifically is that I have >100 sets of sequences, and for each set I construct a position weight matrix, which I then use to score against all sets merged together. I have several scoring schemes so I'd like to determine which ones give me the best classification. For this, I use cross validation: split my data into 5 for each set, train my pwm with 4/5ths of the data and test it on 1/5th. Pool the results from 5 runs, and plot an AUC.

$\endgroup$
3
  • $\begingroup$ Do you care to tell us how many independent samples (cases) you have? $\endgroup$ Nov 15, 2012 at 22:32
  • $\begingroup$ Please explain the relevance of ROC curves in this context. $\endgroup$ Nov 15, 2012 at 22:41
  • $\begingroup$ Omar, I have a set of sequences for which I run a PWM on. I'd like to generate a ROC curve, but am having difficulty in "setting it up". May we move this to chat or I can send you an email if you can help me out? $\endgroup$
    – masfenix
    Jan 26, 2015 at 21:05

2 Answers 2

6
$\begingroup$

There is more to k-fold CV than you do. In essence, the idea of using those crazy splits instead of simply making a few random subsamples is that you can reconstruct the full decision and compare it with original just like you might have done with a predictions on a full train set.

So, sticking to a full k-fold CV mechanism, you just have to merge the predictions from all folds and calculate the ROC for that -- this way you get a single AUROC per model.

However, note that just having two numbers and selecting greater is not a statistically valid way of making comparisons -- without spreads of those two you can't invalidate the hypothesis that both accuracies are roughly the same. So if you are sure you want to do any model selection, you'll need to get those spreads (for instance by bootstrapping the k-fold CV to actually get several AUROC values per classifier) and do some multiple comparison test, probably non-parametric.

$\endgroup$
3
  • $\begingroup$ Thank you! I never thought of that way of combining the data. Would it give me the same effect as averaging? I understand its not statistically valid to compare AUC values. I will probably use DeLong's method. However, with so many models, each comparison takes quite a while. $\endgroup$
    – Omar Wagih
    Nov 16, 2012 at 16:48
  • $\begingroup$ Currently, I'm using the pROC package in R, and the roc.test method. But the wait time is too long $\endgroup$
    – Omar Wagih
    Nov 16, 2012 at 16:54
  • $\begingroup$ @mbq - I am very interested in your last sentence "probably do some multiple comparison test, probably non-parametric". Would you mind elaborating on that? $\endgroup$ Jul 17, 2013 at 13:57
1
$\begingroup$

Just to chime in the @mbq's multiple testing: if you want to compare each of 120 models with each other, that is 7140 comparisons!

You may want to reduce the number of models beforehand by your expert knowledge of the problem. Or include a (few) models that give you baseline performance (constant prediction, random prediction) to test where they fall in the range of all those other models.

Also, make sure you have an independent test set left if you want to report the final performance of the chosen model. Data-driven optimization means that information from the test samples enter your final model as you choose a model that performs well for these (CV) test sets.


Update:

  • Omar, take Frank's hint seriously and read about other performance measures that are better suited.

  • If you decide to stay with AUROC, make sure you calculate them in a range of sensitivities and specificites that are sensible for your application.

  • as mbq says, calculate the spread of your AUROC values of one model, and then think whether you have any chance to identify a good model out of 120 models with 100 independent test cases.

  • In any case: If you want to be able to claim a performance of the final model, you need to test that with a completely independent test set. Samples that were tested for parameter optimization or model selection are not independent any longer. And you should report the uncertainty on this final

$\endgroup$
4
  • $\begingroup$ I can't reduce the number of models. Each model has to be tested. Each model classifies different things. More specifically, my models are position weight matrices which represent different proteins. Thus, I need a proxy of how well the PWM picks up its target in a set of possible targets. $\endgroup$
    – Omar Wagih
    Nov 16, 2012 at 16:50
  • $\begingroup$ @Omar: different things being different input variables, i.e. you want to do a variable selection? Can you explain your application a bit more in the question? $\endgroup$ Nov 17, 2012 at 10:33
  • $\begingroup$ I modified the question. Does this make sense? $\endgroup$
    – Omar Wagih
    Nov 21, 2012 at 16:54
  • $\begingroup$ The relevance of ROC curves for this problem still eludes me, and it also seems there is a problem with the use of improper accuracy scoring rules. $\endgroup$ Nov 21, 2012 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.