Typically when one does cross-validation, one fits $K$ models of the same form over a grid of a hyperparameter. One selects the hyperparameter that minimizes the prediction error out-of-sample averaged over the $K$ folds. With that estimated optimal hyperparameter, one re-fits one's model to the whole dataset. That model is then used for prediction.
This works well for regularized linear regression, because there is usually (always?) a minimum to the out-of-sample prediction error with respect to the hyperparameter.
But some algorithms don't have this guarantee, and/or they are just really noisy, and/or there are multiple hyperparameters, and/or the structure of the data means that some folds might be distributed differently than other folds.
For example, I'm working with panel data with multiple observations of individuals over years, and tuning my model to predict new years out-of-sample. So I've got $N$ observations and $n$ years, with $N>>n$. Different years have fairly different realizations of the covariates, which leads to optimal hyperparameters being pretty different in different folds. Plus I'm fitting neural nets, which are really noisy and where convergence to a global minimum is impractical and/or ill-advised.
So lately I've been combining bagging with cross-validation. The algorithm is:
- Divide the data into $K$ folds
- For each fold, fit the model to the not-$k$th subset over the grid of the hyperparameter. Determine the optimal hyperparameter for that fold by predicting against the $k$th data subset
- Save $K$ different models, each with different hyperparameters
- At prediction time, $\hat y = E[m_k(X)]$ where $m$ is the selected model for that fold.
I get fairly good performance in my application, as measured by out-of-bag error.
The approach seems fairly intuitive, but I've invented it myself (i.e.: I'm not aware of others who have studied this approach). Is there reason to believe that it has drawbacks in certain settings that I should be cautious about? Or are there other approaches to this general class of problems?
Edit: As frequently happens, a linked "related" question provides some insight: A comment on this question links to this paper, which argues -- in the context of many many bootstrap samples -- that one should select the level of the hyperparameter using cross-validation, and given that hyperparameter go back and re-do the bagging. It isn't immediately clear how well this would work in the context of only $K$ bootstrap samples (which are the same as the $K$ folds.)