3
$\begingroup$

I have a few gaps in my understanding of K fold CV.

What I understand: K fold CV is meant to be a model validation technique, the idea is that you subset your data into K subsets, train your data on K-1 of those subsets and test on the Kth subset, repeating this process K times. The final result is the average error over each trial, the CV error.

K fold CV is NOT meant to compare different instances of the same model. For example, if the model to be fit is linear, then you cannot compare linear model 1 with 4 coefficients to linear model 2 with 5 coefficients. You can however compare a linear model CV error with a glm CV error.

If that is the case, then I am confused about fitting a model. Lets say I want to fit a poisson GLM. I use 5 fold CV. On each fold the result of training the model gives me differing choices of parameters and differing numbers of parameters.

For example, on the first fold, the GLM chooses $X_1, X_2$, on the second fold, it chooses $X_2,X_3,X_4$. So at the end of the process, which model do i actually choose?

$\endgroup$
3
  • 1
    $\begingroup$ You can view CV as a way to check the performance of a method. The coefficients of each "fold-models" is not mportant in the least, what is important is everything else : the way you fitted your model, the transformation of your data, the value of your parameters, etc... (for example the $\lambda$ in a LASSO regression of the $\alpha$ of an elastic net). And once you did check the validity of that method through CV you run that method on the whole training set. $\endgroup$ – Riff Sep 27 '16 at 8:11
  • $\begingroup$ @Nicolas if the coefficients of the 'fold models' are not important, then what is the optimal way to find the coefficients for the final model? For the example of a lasso regression, I can use cross validation to find the optimal $\lambda$, is it then sufficient to fit the lasso on the entire data using the optimal lambda? What if i wanted to compare two competing methods? $\endgroup$ – WeakLearner Sep 27 '16 at 11:10
  • $\begingroup$ Yeah that's right ! Once you have found the optimal $\lambda$ value you fit a model on your whole dataset with that value. If you wish to compare two methods you then need to define an evaluation criterion (for example MSEP or misclassification rate) and compute that criterion for your whole dataset using CV, you finally choose the method that optimize your criterion $\endgroup$ – Riff Sep 27 '16 at 12:22
3
$\begingroup$

Cross validation is often used to tune complexity. In your example, some kind of regularisation is (presumably) driving the selection of a different parameter set. Two popular algorithms where CV is used in this way very often is glmnet, which tunes over its regularisation penalty $\lambda$, and boosted decision trees, which tune over the number of trees. In principle, though, CV can be used to tune for a lot of different things like model hyperparameters or model structure.

Once the model structure or complexity parameter is determined, one generally trains a model on the entire training data and selects the optimal parameters as chosen by cross validation.

Tune with CV, then refit across the full dataset. The models that are trained on the $k$ reduced-size datasets are generally not used for anything post-tuning.

$\endgroup$
2
$\begingroup$

You could use K-fold CV in this manner:

  1. Create K-folds of your dataset.
  2. Identify different candidate algorithms you want to evaluate to build your model. Lets assume you have 2 algorithms to evaluate - linear regression, logistic regression.
  3. For each one, run parameter selection training runs for each of K-folds and evaluate their accuracy on the held-out Kth fold. Also tune the hyper-parameters for these models in the same training runs - e.g. L1 or L2 regularization parameters. Lets assume you have identified 2 sets of parameters for each algorithm - 2 for linear regression and 2 for logistic regression.
  4. For each of these 4 candidate models, evaluate their performance on the K-folds. Assume you prefer selecting the one with lowest variance of True Positive accuracy - let this be model #3. Other selection criteria could be to select the model with highest average TP accuracy, etc.
  5. Fit model #3 on the entire training data with the selected hyper-parameters to get the final model.

Its not recommended to tune the model on the entire training data (Step #5 , e.g. selecting the parameters, hyper-parameters, etc.) because if you do so, you won't be able to check if you made a robust model which will perform as expected with new data. The Kth held out fold lets you do that and fix any mistakes during model building.


Edited to explain queries in comments:

Parameter imply all the input features used by the model and Hyper-praameters mean all the other configuration settings of the algorithm used to produce the optimal fit. For example, if we're fitting a neural network , selecting the variables x1, x2, x3 as the best inputs to produce an estimate y implies we're selecting these 3 parameters for the model. Hyper-parameters would be the no. of units, no. of hidden layers and the learning rate of the neural network.

When we reach step #5, we will have the final list of input parameters which had achieved the best results in cross-validation, and the hyper-parameters which had given the best results. The K-folds and the coefficients learned for each fold are discarded. Then, we use the entire training data, pick our final list of parameters and run a final training using the selected hyper-parameters to get the model fit, or, coefficients that collectively represent the completed "model".

$\endgroup$
9
  • $\begingroup$ I'm quite confused with your step 3. what does regularisation have to do with it? $\endgroup$ – WeakLearner Sep 27 '16 at 5:18
  • $\begingroup$ Regularization is always recommended, however you can choose to ignore it. I mentioned it to include covering the hyperparameter selection process. Most algorithms would require selecting a few hyper-parameters to fine tune their accuracy. For example, you would need to do this for all tree based, SVM, neural network based and all ensemble based methods. $\endgroup$ – Sandeep S. Sandhu Sep 27 '16 at 5:25
  • $\begingroup$ so you are saying for my K folds, fit a regularised Poisson GLM. Let's say this returns K poisson GLMs each with different variables chosen. Then take each of the K models and evaluate their performance on the K test sets and pick the with lowest variance. Choose the best model of those K poisson glms, then fit the model again on the entire training set? Doesn't this defeat the purpose? as fitting on the entire training set might give us a model with completely different no. of parameters to the model that minimized variance in the previous step? $\endgroup$ – WeakLearner Sep 27 '16 at 5:29
  • $\begingroup$ The last step #5 is meant to finish the model training using the already selected set of parameters (#4). You do not need to re-select parameters in step #5. Selecting and trying our model performance on K-folds is intended to expose the model to unseen new data, which is what will be experienced in production when it is deployed to perform on new data - the Kth held-out fold mimics new data. Also, I mentioned selecting the model with lowest variance as an example of what I prefer, you may select a model with highest average TP rate, or whatever best fits your particular case. $\endgroup$ – Sandeep S. Sandhu Sep 27 '16 at 5:44
  • $\begingroup$ Continuing in new comment since space ran out in previous comment. You should'nt perform model tuning on the entire training data (e.g. selecting the parameters, hyper-parameters, etc.) because if you do so, you wont be able to check if you made a robust model which will perform as expected with new data. The Kth held out fold lets you do that and fix any mistakes during model building. $\endgroup$ – Sandeep S. Sandhu Sep 27 '16 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.