3
$\begingroup$

I understand Cross Validation is used to parameter tuning and finding the machine learning model that will generalize well on the test data/

Leave one out cross validation: One data point is left out (validation) and trained on rest of the data points. This is done iteratively and then results are averaged out. But how do we find the best model when the results are average out ?

K-Fold we say K = 5 , 4 fold data is trained on and 1 fold data is validated and the results again are average out. Again how do we know which is the best model here ? Or do we choose a hyper parameter and do K-Fold here to see best hyper parameter in the model ?

Lastly, what is the difference between grid search CV and K-Fold ?

$\endgroup$
1
$\begingroup$

Cross validation is used to either compare different models or to tune the parameter. For the case of the parameter tuning, you can consider the models with different parameters to be unique models that you want to test. For both cases, the $MSE_i$ are averaged out for each model, and then compared to $CV$ parameter in different models. These $CV$ parameters cannot be used in absolute sense, they only make sense when compared to $CV$ parameters from the same data, same kind of cross validation, and different model (think of this like you would think of AIC and BIC). Once you have the $CV$ parameters from different models, you can choose the appropriate model (or parameter) by picking the smallest number (smaller the average $MSE$ the better).

Grid search cross validation is usually employed for parameter tuning, while K-fold is more general approach. Grid-search is a way to select the best of a family of models, parametrized by a grid of parameters.

$\endgroup$
  • 1
    $\begingroup$ @v-aslanyan Thanks for the answer. Now just for clarification 1) Leave one out cross validation: The steps are train on n-1 points and test on 1 left out point. Keep a note of the error on the test point. $\endgroup$ – Sheldon Aug 26 '18 at 3:31
  • 1
    $\begingroup$ In the above scenario how do we find out which is the best model ? Or are we also changing a tuning parameter λ to find least error ? 2) K-Fold : I again know that K-1 fold are training data and 1 left out fold is test data. Do we simply just take K = 1 to say 11 folds and find which K value performs best ? Or Do we again use some hyper parameter λ to find least error 3) What confuses me is using K-Fold where K = 5 and tuning for hyper parameter λ is that nest cross validation error ? 4) If 3 is correct that what is grid search ? Is that for 2 hyper parameters ? say depth and λ $\endgroup$ – Sheldon Aug 26 '18 at 3:35
  • 1
    $\begingroup$ @Sheldon Lets assume we have a data with 12 potential predictors. First we try linear regression with 8 and 10 and 12 predictors. For linear regression no parameter tuning is needed because of the nature of the model, so we can either use K-fold or LOOCV to find the best model out of those three. The best model here is the one with the lowest $CV=\sum_i MSE_i$. Now imagine we try a Ridge regression. Parameter tuning is a common practice for Ridge regression, and here is when you need to tune your $\lambda$. You can again choose your LOOCV or K-fold and do a grid search for $\lambda$ and pick $\endgroup$ – V. Aslanyan Aug 27 '18 at 5:17
  • 1
    $\begingroup$ the best model according to CV defined above. 2) choosing K does not test how good your model performs. Your choice of K does not affect the model itself, so you should keep your K constant among your 3 linear models and then pick the best model. The hyper parameter is only applicable to some models, so it really depends on your case. In this example of linear models, you just pick K=1 and then test it among your 3 linear models. 3) you can do grid search among your potential hyper parameter in the ridge case by using 5 fold cross validation. your program would go through each hyper $\endgroup$ – V. Aslanyan Aug 27 '18 at 5:32
  • 1
    $\begingroup$ parameter, fit a model, and perform a 5 fold crossvalidation. it will calculate a CV number for each of them and automatically choose a $\lambda$ for which this CV is the minimum. For better understanding of grid search and cross validation please have a look here $\endgroup$ – V. Aslanyan Aug 27 '18 at 5:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.