I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind: 1) Is the high variance in the testAUC simply due to randomness of holdout data selected? 2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC? 3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind:

1) Is the high variance in the testAUC simply due to randomness of holdout data selected?

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind: 1) Is the high variance in the testAUC simply due to randomness of holdout data selected? 2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC? 3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind: 1) Is the high variance in the testAUC simply due to randomness of holdout data selected? 2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC? 3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.