I have a set of true positive (TP) values which are used to train a model.
I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and 1/5th for testing)
I repeat this using different 1/5ths as the test set. For each run, I have a large set of mixed true positives / true negatives which I use my trained model to attempt to classify. I then obtain an ROC curve. This is done for each run of the cross validation (i.e. I end up with 5 ROC curves)
I then average the AUC and return it.
My problem:
I have two methods of classification: call them method A and method B. for each method, I get 5 ROC curves. How can I determine which method gives me a better ROC if I have more than one ROC for each?
I know computing the AUC and averaging for each method, then comparing averaged AUCs is NOT a good approach.
Note: I have more than 1 model (roughly 120). I just explained in terms of one model for simplicity. So I have 120 models, each one having classified the data using method A and method B, and for each method A & B there are 5 ROCs from cross validation.
Edit
My problem more specifically is that I have >100 sets of sequences, and for each set I construct a position weight matrix, which I then use to score against all sets merged together. I have several scoring schemes so I'd like to determine which ones give me the best classification. For this, I use cross validation: split my data into 5 for each set, train my pwm with 4/5ths of the data and test it on 1/5th. Pool the results from 5 runs, and plot an AUC.