I am using 10-fold cross-validation to build a classifier (logistic regression). For the same data set (which is ~2000 rows), I randomly hold out 10% and run 10-fold C.V. on the remaining 90% for a range of $\lambda$ (ridge) and $\alpha$ (elastic net) values. I run the same model building procedure several times (say 5 or 10), each time randomly selecting a different holdout set to do testing on. Here is a typical run for one of my models with training and testing AUC:

trainAUC            testAUC
0.7789858700489541  0.614762386248736
0.7762811027773526  0.6525764895330113
0.7744834303471625  0.6282312925170068
0.7710854322029923  0.6379084967320261
0.7703260594826858  0.7139756944444444
0.7678740678991903  0.650191570881226
0.7590972626674432  0.7620200622621931
0.7571686726448225  0.750197628458498
0.7492527543821031  0.58
0.7335912555731339  0.7116920842411039

You can see that the training AUC is very consistent, but the testing AUC varies widely from a low of 0.58 to a high of 0.76. This raises a few questions in my mind:

1) Is the high variance in the testAUC simply due to randomness of holdout data selected?

2) If I was forced to select a single model, should I select the model with highest training ROC or test ROC?

3) Would it make sense to create an ensemble classifier which uses each model to make predictions and then averages the predictions?

Note that I am not simply asking for the definition of cross-validation. I know what it is, and I am using it correctly. This is more about model comparison, not parameter selection.

  • $\begingroup$ Possible duplicate of Cross-Validation in plain english? $\endgroup$ – Sycorax Sep 29 '16 at 18:09
  • $\begingroup$ I'm not asking what cross-validation is. You may be misunderstanding my question. Each result in the table I listed is from a different model, each one of which is created using cross-validation. My question is more about model selection, not parameter optimization. $\endgroup$ – thecity2 Sep 29 '16 at 18:19

1) Yes, that's the reason.

2) Not necesarily, I hope it's not a mediocre answer if I tell you to trust your software. If you do not trust your software, you should calculate an average.

3) Same as (2), I think

  • $\begingroup$ I think you're misunderstanding my question. These are not the results from one run of cross-validation. The above results are from repeated runs of cross-validation each of which is 10-fold. So the "software" (Spark in this case) has already made a selection, 10 times in this case, and the results above are for that selection (i.e. 10 different models each selected by CV). $\endgroup$ – thecity2 Sep 29 '16 at 19:11
  • $\begingroup$ Does spark return a vector of size 10 with the estimated parameters? $\endgroup$ – Juan Esteban de la Calle Sep 29 '16 at 19:12
  • $\begingroup$ It only returns a "training AUC" for the best model selected by CrossValidator. This is what I have listed in the left-hand column for 10 models trained in this way (i.e. me running a script 10 times). The test AUC is me taking the best model from each cross-validation run and testing it on a hold out set (10%) that was randomly selected before cross-validation. $\endgroup$ – thecity2 Sep 29 '16 at 19:14
  • $\begingroup$ This question I asked (and got no responses) about AUC in Spark is related to the comment above (and perhaps sheds some light on my question here): stackoverflow.com/questions/39516668/… $\endgroup$ – thecity2 Sep 29 '16 at 19:27
  • 1
    $\begingroup$ Turns out there's also a bug in PySpark that has been fixed that relates to the way the average test metric was being calculated (it always looked weird to me, now I know why!). issues.apache.org/jira/browse/… $\endgroup$ – thecity2 Sep 29 '16 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.