# Should I use training or testing AUC for selecting best classifier?

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039


You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind:

1) Is the high variance in the testAUC simply due to randomness of holdout data selected?

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

• Possible duplicate of Cross-Validation in plain english? – Sycorax Sep 29 '16 at 18:09
• I'm not asking what cross-validation is. You may be misunderstanding my question. Each result in the table I listed is from a different model, each one of which is created using cross-validation. My question is more about model selection, not parameter optimization. – thecity2 Sep 29 '16 at 18:19