# Should I use training or testing AUC for selecting best classifier?

I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039


You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind:

1) Is the high variance in the testAUC simply due to randomness of holdout data selected?

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

• Possible duplicate of Cross-Validation in plain english? – Sycorax Sep 29 '16 at 18:09
• I'm not asking what cross-validation is. You may be misunderstanding my question. Each result in the table I listed is from a different model, each one of which is created using cross-validation. My question is more about model selection, not parameter optimization. – thecity2 Sep 29 '16 at 18:19
• Logistic regression is not a classifier. It is a direct probability estimation model. And AUROC (concordance probability) is not sensitive enough for your purposes. – Frank Harrell May 30 '19 at 23:58

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

How can you select a model if its the same parameter set but just a different iteration? (high variance)

If you run repeated 10-fold CV ("5 or 10 times") and get different AUC values for the same parameter set, then a fair estimation is the worst outcome. Cross-validation in itself is already a pessimistic estimation of the model trained on the entire data set (that was used for CV). Select 0.58 (the lowest test AUC) - selecting the best train or test AUC is probably over-optimistic.

If 0.58 is not good enough, then the model must be made more robust to noise - trading off less variance for the cost of more bias - that makes selecting a model easier. At some point you need a different hold-out set to test since you are somewhat optimizing your testAUC if you keep reiterating.

1) Yes, that's the reason.

2) Not necesarily, I hope it's not a mediocre answer if I tell you to trust your software. If you do not trust your software, you should calculate an average.

3) Same as (2), I think

• I think you're misunderstanding my question. These are not the results from one run of cross-validation. The above results are from repeated runs of cross-validation each of which is 10-fold. So the "software" (Spark in this case) has already made a selection, 10 times in this case, and the results above are for that selection (i.e. 10 different models each selected by CV). – thecity2 Sep 29 '16 at 19:11
• Does spark return a vector of size 10 with the estimated parameters? – Juan Esteban de la Calle Sep 29 '16 at 19:12
• It only returns a "training AUC" for the best model selected by CrossValidator. This is what I have listed in the left-hand column for 10 models trained in this way (i.e. me running a script 10 times). The test AUC is me taking the best model from each cross-validation run and testing it on a hold out set (10%) that was randomly selected before cross-validation. – thecity2 Sep 29 '16 at 19:14
• This question I asked (and got no responses) about AUC in Spark is related to the comment above (and perhaps sheds some light on my question here): stackoverflow.com/questions/39516668/… – thecity2 Sep 29 '16 at 19:27
• Turns out there's also a bug in PySpark that has been fixed that relates to the way the average test metric was being calculated (it always looked weird to me, now I know why!). issues.apache.org/jira/browse/… – thecity2 Sep 29 '16 at 19:38

I wouldn't run 10-fold CV and test with 10% of randomly selected objects and train with the remaining 90%. What if you randomly select the next 10% for testing and one of the objects in the first 10% of tested objects was selected again? As you go through the 10 folds in order to train with the remaining 9 folds left out of testing on each fold, objects are not supposed to appear in other folds in duplicate. The 0.632 bootstrap CV accuracy approach does perform resampling with replacement, but it's not 10-fold CV.

You are supposed to:

1. Shuffle all the objects (permute them randomly, i.e. re-order)
2. Assign the re-ordered objects to the 10 folds. (if the sample size is $$n=100$$ and the first 10 shuffled object ID's are 23,7,82,43,66,17,98,36,11,76, they all get assigned into fold 1, the next 10 objects are assigned to fold 2, etc.)
3. Select fold 1 for testing and folds 2-10 for training,
4. Repeat step 3 until you have tested objects in all 10 folds
5. Go to step 1, and repeat this "re-partitioning" step 10 times.

The above is called "ten 10-fold CV", for which you randomly shuffle (permute, re-order) all objects and re-assign them into 10 folds, and the go through the 10 folds for testing.

A problem with the results you are providing is that you only ran 10-fold CV one time, and never randomly shuffled the order of all objects before re-assigning them into 10 folds again, and trained/tested with the 10-folds.

There should be no random selection of objects during 10-fold CV, as the only thing that is random is the permuted (random) order of all objects before assigning them into folds. This is just called "shuffling", like shuffling 52 cards in a deck of cards.

Another thing you're supposed to be doing is comparing results from 2-fold, 5-fold, 10-fold, LOOCV, and CVB. (where LOOCV is leave-one-out CV, and CVB is bootstrap bias CV).

Your OP basically says: 10-fold CV was ran once, so what's the best classifier? The answer depends on the data. Of course you're not supposed to use training AUC to evaluate classifiers.