2
$\begingroup$

I am learning how to perform K-fold CV and I have a few questions about the method. This is an excerpt from the website: https://towardsdatascience.com/cross-validation-in-machine-learning-72924a69872f.

In K Fold cross validation, the data is divided into k subsets. Now the holdout method is repeated k times, such that each time, one of the k subsets is used as the test set/ validation set and the other k-1 subsets are put together to form a training set. The error estimation is averaged over all k trials to get total effectiveness of our model. As can be seen, every data point gets to be in a validation set exactly once, and gets to be in a training set k-1 times. This significantly reduces bias as we are using most of the data for fitting, and also significantly reduces variance as most of the data is also being used in validation set. Interchanging the training and test sets also adds to the effectiveness of this method. As a general rule and empirical evidence, K = 5 or 10 is generally preferred, but nothing’s fixed and it can take any value.

Notice that the much of the explanation done here is about the error itself.

Let's say that I am performing a linear regression on sets of data. (This is the model I want to have.) Additionally I have 10 sets of data where I fit a model on.

If I decide to have $k = 5$ (from this text), I will have 2 sets of data per $k$. So I can fit a model on these 2 sets, and test the data with the other sets and record their errors.

My question: So I will need to repeat this procedure with other sets of data (so that each $k$ will become a validation set exactly once), what happens to my model? Which model do I choose?

Insights please.

$\endgroup$
  • 1
    $\begingroup$ @cgo- You might want to share if the provided answer was helpful or not. And help improve it. $\endgroup$ – naive Nov 5 '18 at 11:03
1
$\begingroup$

If I decide to have k=5 (from this text), I will have 2 sets of data per k. So I can fit a model on these 2 sets, and test the data with the other sets and record their errors.

You would be training the model on $4$ sets and estimating the prediction error on remaining $1$ set.

For standard linear regression i.e OLS, there is no Hyperparameter. If you are doing OLS, then I am afraid you can only use cross-validation to estimate the prediction error for the given dataset. You really don't have a choice of models to select from. This is because you don't have models indexed by tuning parameters.

But, let's say you wanted to perform ridge regression or LASSO then you have lambda i.e. shrinkage factor as a Hyperparameter. In this case if you use different shrinkage factors, you want to find the best shrinkage factor among them.

So, you would perform cross validation for each shrinkage factor and get the prediction error estimate(average across the validation sets) for each. Then select the model(shrinkage factor) with the least prediction error estimate.

$\endgroup$
0
$\begingroup$

The purpose of cross-validation is to have a statistically sound estimate of how well can a specific model perform for the specific problem (and dataset).
It is not clear what are the 10 datasets you mention in your question. Note that it is considered a good practice to shuffle your data before splitting it for cross-validation.
Yes - you need to repeat the procedure of train (on (k-1) folds) and test (on 1 fold) k times. Then you average the error over the test folds. The resulting error can be used to compare among different models (for example different types of regression, or different hyperparameters' values).
After you choose a specific model (e.g., based on the cross-validation results), you can train it on the entire dataset and this model can be used for production.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.