2
$\begingroup$

I am confused about the concept of cross validation and its usage.

As I read about cross validation before, it is a way of validating a model. I did cross validation in my project (developing different regression models on a dataset, model validation and finally choosing best model). Cross validation just give me a model but no statistical criteria that shows ability of the model. By the way each time that I run cross validation (in R program) the result is different because train dataset is changed. For selecting the best model, I calculate AIC for the model obtained by cross validation but now I think it is wrong. Because first, train data used for each model is different, and second, even for a certain model, AIC will change by repeating cross validation.

Lately I read that cross validation is used for selecting the best model! All these confused me about utility of cross validation and the way of interpreting the result.

Could you please help me on figuring out all these?

$\endgroup$

2 Answers 2

1
$\begingroup$

Cross-validation is essentially a method to estimate generalization performance.

I'm going to assume you are considering k-fold cross-validation, which is the most common type (I guess). It can be employed during both model selection and what some might call validation:

  • model selection procedures can use cross-validation to estimate performance of a model built using certain hyperparameters/structure and then select the model with the best estimated performance. Cross-validation itself does not select a model, but it can be used to estimate the generalization performance of a model.
  • in nested cross-validation two layers of cross-validation are used:
    1. the outer layer estimates generalization performance of a full modeling approach, including model selection
    2. the inner cross-validation procedure is used for model selection, as explained previously
$\endgroup$
2
  • $\begingroup$ Yes, I used 5-fold cross validation. I need to know if the result of cross validation changes by rerunning it, so how we can trust its result? In other word, should we run cross validation just one time ant trust the randomly selected train data and the model obtained? If yes, I think it does not make sense because when I calculate AIC of the model, sometimes it’s very good and could be lowest AIC comparatively other models and sometimes it is high and not good at all. $\endgroup$ Aug 4, 2014 at 8:42
  • $\begingroup$ @user3713988 It is normal that the outcomes of cross-validation are somewhat stochastic (though they should be within reasonable bounds). If your models exhibit high variance, symptomized by very large performance differences in cross-validation, you should consider (1) changing modeling strategy, as it seems unreliable (2) performing iterated cross-validation to get more stable performance assessments. I suggest option 1. If you go for option 2 without changing strategy, be aware that your model may perform well below the cross-validated performance estimate if you are unlucky. $\endgroup$ Aug 4, 2014 at 9:39
0
$\begingroup$

I think the "model validation" part of cross-validation is that it allows you get a good approximation of the performance of the model (expressed in a measure of goodness-of-fit like mean squared prediction error or AIC) outside of the training set.

Of course these performance values by themselves do not tell you if your model is correct, usually they can only tell you if your model is better than some other given model. However, getting a very small cross-validated mean squared prediction error is at least a good sign that your model is good predictor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.