Should I use training or testing AUC for selecting best classifier?

Question

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind:

1) Is the high variance in the testAUC simply due to randomness of holdout data selected?

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

Answer 1

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

How can you select a model if its the same parameter set but just a different iteration? (high variance)

If you run repeated 10-fold CV ("5 or 10 times") and get different AUC values for the same parameter set, then a fair estimation is the worst outcome. Cross-validation in itself is already a pessimistic estimation of the model trained on the entire data set (that was used for CV). Select 0.58 (the lowest test AUC) - selecting the best train or test AUC is probably over-optimistic.

If 0.58 is not good enough, then the model must be made more robust to noise - trading off less variance for the cost of more bias - that makes selecting a model easier. At some point you need a different hold-out set to test since you are somewhat optimizing your testAUC if you keep reiterating.

Answer 2

1) Yes, that's the reason.

2) Not necesarily, I hope it's not a mediocre answer if I tell you to trust your software. If you do not trust your software, you should calculate an average.

3) Same as (2), I think

Answer 3

I wouldn't run 10-fold CV and test with 10% of randomly selected objects and train with the remaining 90%. What if you randomly select the next 10% for testing and one of the objects in the first 10% of tested objects was selected again? As you go through the 10 folds in order to train with the remaining 9 folds left out of testing on each fold, objects are not supposed to appear in other folds in duplicate. The 0.632 bootstrap CV accuracy approach does perform resampling with replacement, but it's not 10-fold CV.

You are supposed to:

1. Shuffle all the objects (permute them randomly, i.e. re-order)
2. Assign the re-ordered objects to the 10 folds. (if the sample size is $n=100$ and the first 10 shuffled object ID's are 23,7,82,43,66,17,98,36,11,76, they all get assigned into fold 1, the next 10 objects are assigned to fold 2, etc.)
3. Select fold 1 for testing and folds 2-10 for training,
4. Repeat step 3 until you have tested objects in all 10 folds
5. Go to step 1, and repeat this "re-partitioning" step 10 times.

The above is called "ten 10-fold CV", for which you randomly shuffle (permute, re-order) all objects and re-assign them into 10 folds, and the go through the 10 folds for testing.

A problem with the results you are providing is that you only ran 10-fold CV one time, and never randomly shuffled the order of all objects before re-assigning them into 10 folds again, and trained/tested with the 10-folds.

There should be no random selection of objects during 10-fold CV, as the only thing that is random is the permuted (random) order of all objects before assigning them into folds. This is just called "shuffling", like shuffling 52 cards in a deck of cards.

Another thing you're supposed to be doing is comparing results from 2-fold, 5-fold, 10-fold, LOOCV, and CVB. (where LOOCV is leave-one-out CV, and CVB is bootstrap bias CV).

Your OP basically says: 10-fold CV was ran once, so what's the best classifier? The answer depends on the data. Of course you're not supposed to use training AUC to evaluate classifiers.