2
$\begingroup$

I saw a presentation by a student in my department, where he's demonstrating k-fold cross validation using OLS.

He split the dataset $\mathcal{D}$ into k partitions as usual, and trained k OLS models on $\mathcal{D}\setminus \mathcal{D_k}$ and assessed the k-th model on the $\mathcal{D_k}$. So all of this so far is aligned with my understanding.

But then he made the statement that the model with the lowest prediction error is the chosen model, but he didn't train the "chosen" model on the entire $\mathcal{D}$ after performing k-fold CV.

Is this procedure wrong? I thought you're supposed to train the chosen model on the entire dataset after performing cross validation?

Improve this question
$\endgroup$
8
  • $\begingroup$ I have not yet understood the second part. Could you outline the process with an example as simple as possible? $\endgroup$
    – Michael M
    Commented Jun 29, 2020 at 19:34
  • $\begingroup$ @MichaelM Unfortunately, I don't have his presentation slides, so everything I'm stating is from my memory of the presentation. When I read my question and think about what he did, I feel like it doesn't make sense. He basically suggested that he found $k$ sets of $\hat{\beta}$ OLS parameters. For future modeling purposes, he chose the $\hat{\beta}$ that gave the smallest prediction error observed during the k-fold validation. $\endgroup$
    – user5965026
    Commented Jun 29, 2020 at 19:41
  • $\begingroup$ This approach is flawed. K-Fold CV is not for model selection, it is for model evaluation. The results can be used to select between multiple models that you evaluated individually, but for a given model, the various folds are just used for evaluating the models out of sample performance. The estimated parameters and metrics of each fold are not used for anything $\endgroup$
    – Ryan Volpi
    Commented Jun 29, 2020 at 19:53
  • $\begingroup$ That sounds very wrong then! $\endgroup$
    – Michael M
    Commented Jun 29, 2020 at 19:54
  • 1
    $\begingroup$ @user5965026 You are correct and I believe I worded that poorly. K-Fold CV can be applied to estimate the performance of multiple models, or the same model with different hyperparameters, and that data can be used for model selection. The distinction I wanted to make is that K-Fold CV itself is not a model selection algorithm or procedure like grid search. $\endgroup$
    – Ryan Volpi
    Commented Jun 29, 2020 at 20:15

1 Answer 1

Reset to default
2
$\begingroup$

The typical procedure is to choose the best hyperparameters, which OLS doesn't have, based on cv performance and train the model on the entire training set (as you expected to be). The procedure that the student employs is actually choosing a dataset. And, it increases the variance because if any of the folds have a bad partition, you may end up choosing that one as your model. Instead of an averaged performance, it relies on a specific dataset partition. This is especially why we'd generally prefer cross-validation while tuning HPs instead of a holdout set if not computationally expensive.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Would you ever use CV for plain OLS? I think in a way, you can view the number of explanatory variables / features in OLS as hyperparameters, and use CV for feature selection of a subset of the original features? When you say "increases the variance," what are you saying increases in variance? There seems to be some confusion on this based on other posts on k-Fold Validation that I've read. But those seem to be focused on variance increasing in the prediction error as $k$ increases. $\endgroup$
    – user5965026
    Commented Jun 29, 2020 at 20:02
  • 2
    $\begingroup$ "The procedure that the student employs is actually choosing a dataset." Hmm this seems to be exactly what he did under the hood. He found a training dataset and testing dataset that gave a $\hat{\beta}$ that achieves the best prediction error. That doesn't seem to serve as practical use. $\endgroup$
    – user5965026
    Commented Jun 29, 2020 at 20:04
  • 1
    $\begingroup$ Here, by increasing the variance, I mean increasing the possibility of overfitting. Yes, you can k-fold on feature subsets. $\endgroup$
    – gunes
    Commented Jun 29, 2020 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.