I have a high dimensional regression problem where I used glmnet to solve this. A nested CV scheme is used. In the inner CV loop (10x5 fold) a grid search is done to find the optimal hyperparameters for glmnet. The outer CV loop (10x5 fold) is used to estimate performance of the regression with the previously found optimal parameters. However, when I apply this scheme something strange happens: the RMSE in the outer loop is way better than in the inner loop. This is absolutely counterintuitive because one would expect that the model works well in the inner loop because it was trained directly on these data and the model should be worse in the outer loop because it has "never seen" this data beforehand. Has anyone an explanation? My data set has 172 instances in total and 474 variables.
-
$\begingroup$ Just for my own understanding, what does 10x5 refer to? 10 iterations on 5 folds? Is each fold left out twice? $\endgroup$– shadowtalkerCommented Mar 14, 2015 at 11:50
2 Answers
You probably have the learning curve working in your advantage, ie. you can train better models given more data. Even though you are optimizing hyperparameters, the best performance in inner CV can be consistently lower because after the inner CV you get 20% more data points to train the model you evaluate in the outer CV. This depends on whether or not you have reached the required number of data instances to make the learning curve stagnate. You appear to be in a region in which it is still steeply increasing.
You will probably get a smaller difference if you use 5x10-fold CV in the inner procedure.
It should be noted that some parameters you obtain in this way might be suboptimal. Regularization parameters, for instance, are affected by sample size. If your data set is small (and yours is), you may misestimate the required regularizers because a 20% difference in sample size can be considerable. This can be remedied by using more CV folds in the inner procedure (ie. increase $k$ in $k$-fold CV).
-
$\begingroup$ Why would more folds help? Won't that just increase the disparity? $\endgroup$ Commented Mar 14, 2015 at 11:54
-
$\begingroup$ @ssdecontrol because when you use more folds, the difference between the sizes of your cross-validation training set and total training set is smaller ($(k-1)/k$). $\endgroup$ Commented Mar 14, 2015 at 13:55
I checked how the results are affected when changing from 5x10-fold CV to 10x5-fold CV:
10x5-fold CV, Train-RMSE: 10.73, Test-RMSE: 5.15 5x10-fold CV, Train-RMSE: 11.28, Test-RMSE: 5.97
So, increasing the number of folds doesn't seem to help much in this case, unfortunately. The problem of almost doubled error in the training set still exists.
-
1$\begingroup$ Are you the OP? If so, please merge your accounts. Then you'll be able to edit & update your own question, & comment on your own thread. You can find out how in our help center. $\endgroup$ Commented Mar 16, 2015 at 13:56