Suppose you perform cross-validation to obtain an optimal value for some vector of hyperparameters $\lambda$.

You ultimately want to predict some new observations $y_\mathrm{query}|X_\mathrm{query}$.

It seems that you have at least three choices on how to proceed:

  1. Estimate the model parameters $\hat\theta_i$ on each cross-validation training sample $i=1,\dots,n$, with the optimal $\lambda$, then average these values to obtain a final estimate, $\hat{\bar\theta}:=\frac{1}{n}\sum_{i=1}^n{\hat\theta_i}$. Use these averaged estimates $\hat{\bar\theta}$ to perform the required prediction, $\hat y_{\mathrm{query},\hat{\bar\theta}}:=\mathbb{E}[y_\mathrm{query}|X_\mathrm{query},\theta=\hat{\bar\theta}]$.
  2. Estimate the required predictions $\hat y_{\mathrm{query},i}$ on each cross-validation training sample $i=1,\dots,n$, with the optimal $\lambda$, then average these values to obtain a final prediction, $\hat {\bar y}_\mathrm{query}:=\frac{1}{n}\sum_{i=1}^n{\hat y_{\mathrm{query},i}}$.
  3. Using the optimal $\lambda$, re-estimate the model on the entire sample, to obtain $\hat\theta_*$. Use these parameters to perform the required prediction, $\hat y_{\mathrm{query},\hat\theta_*}:=\mathbb{E}[y_\mathrm{query}|X_\mathrm{query},\theta=\hat\theta_*]$.

Which of these methods is most common? What are their advantages and disadvantages?


1 Answer 1

  1. This is uncommon and can be quite risky, especially when there are multiple local minima for the cost function. You might even end up in the neighbourhood of a local maximum after averaging them out!

  2. This is similar (not the same since subsets are not bootstrap samples) to ensembling (e.g. random forests). Not very common, and not preferable when training is expensive. It might be useful when you want to reduce the variance of your model, but there are other ways to reduce it as well.

  3. This is the most common approach. Preferable since in the end you can claim one model and it'll be cheaper than training $K$ models. Moreover, in this one, your model sees the whole data together.

  • $\begingroup$ Isn't one argument for 1 or 2 that your optimized hyperparameters will only be optimal for datasets the size of your CV training sample. The full dataset could be much larger, depending on how many folds you have. $\endgroup$
    – cfp
    Commented Jun 17, 2020 at 9:00
  • 1
    $\begingroup$ If number of folds increases, training and validation sets come closer in size. The largest difference is for K=2, where it is halved. But, the point that the datasets differ is valid especially for (1). $\endgroup$
    – gunes
    Commented Jun 17, 2020 at 10:58
  • $\begingroup$ @cfp is the answer clear? $\endgroup$
    – gunes
    Commented Jun 23, 2020 at 15:48
  • $\begingroup$ Yes, thanks. I only didn't tick it as it seemed like there was potentially scope for more to be said, so I didn't want to put off other contributors. $\endgroup$
    – cfp
    Commented Jun 23, 2020 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.