3
$\begingroup$

In this post on stackexchange, the answer states that "The purpose of cross-validation is model checking, not model building." A very good explanation for that is given as follows: "(...) selecting one of the surrogate models means selecting a subset of training samples and claiming that this subset of training samples leads to a superior model."

While this is intuitive, usually

  1. we pick the best performing classifier of the cross-validation, and test this on a further set - usually called the test set. So, we do in fact use cross-validation for building a model in picking a particular one, often in combination with hyperparameter optimization we would chose a specific set of hyperparameters which led to the best validation-set-results. Is this not contradicting the above statement?

  2. if the results of the crossvalidation are just dependent on the particular validation-set, why is model selection justified this way?

$\endgroup$
5
$\begingroup$

we pick the best performing classifier of the cross-validation, and test this on a further set - usually called the test set. So, we do in fact use cross-validation for building a model in picking a particular one, often in combination with hyperparameter optimization we would chose a specific set of hyperparameters which led to the best validation-set-results. Is this not contradicting the above statement?

The situation may be more easily explained if you divide it into a different set of building blocks.

  1. there are techniques to measure model performance, e.g. cross validation, testing a single held-out split of your data, auto-prediction, performing a fully blown validation study.
    They differ wrt. efficiency of data use, systematic (bias) and random (variance) uncertainty, cost/effort etc.
    But technically you can use any of these for the building block "estimate performance"

  2. Optimization, here: choose a good model from a variety of possible models. There are many criteria what a good model is. But one criterion that is widely applicable is predictive performance.
    So if you choose to use this optimization criterion, you then need to choose a suitable way of measuring/estimating the performance of the (surrogate) model you are considering. Doing a full validation study for lots of models isn't feasible, auto-prediction doesn't yield enough information, depending on your data, a single held-out split may be subject to too much random uncertainty, so in the end you settle for cross validation.

So cross validation is used in model optimization just as a light switch is used in a car: you need something to make the light go on and off as needed and you use a solution that already (and primarily) exists outside cars. But thinking of light switches primarily as "something used inside cars" likely doesn't help much understanding how they work and the specific characteristics are. They exist independent of cars, but can be applied inside cars - just as they can be applied inside houses, other machines, etc.
Similarly cross validation exists as a validation (actually: verification) technique, and this technique can be applied for calculating the target functional of your optimization. Or for verification of the optimized model (see below). Or ...

  1. Final Verification: (not really a new building block, but nevertheless necessary) Because we know that variance uncertainty in the target functional used for pick-the-maximum optimization tends to lead to overfitting, we get another independent measurement of the performance of the model we decided on. Again, we have the choice of the methods 1. I may choose to do another cross validation there (aka outer cross validation to distinguish it from the "inner" cross validation inside the optimization) - you may choose to go for the "further test set".

if the results of the crossvalidation are just dependent on the particular validation-set, why is model selection justified this way?

You should not base your decision on the result obtained on a single validation set. Neither a single cross validation surrogate, nor on a single held-out split of your data. Instead, you should judge those results taking into account their uncertainty.
If you suspect that the surrogate models are unstable, i.e. the actual splitting having an influence on the result, you should calculate more splits and directly check this. E.g. by repeated/iterated cross validation.

Keep in mind: The fact that many people happily overfit their models doesn't mean that this is good practice...

$\endgroup$
  • 1
    $\begingroup$ Many thanks for the extensive answer. I understand you saying that we do chose CV to select the optimal model, as we define optimality over predictive performance, something we can quantify with CV. Two questions: I read sometimes that it is better NOT to take the best model from the inner loop. What is done instead? 2. I also read that we need to retrain the chosen model on the entire dataset. Why is that? Why would you not use the same hyperplane (in the case of SVMs for instance) if you assume all your data have the same underlying distribution? $\endgroup$ – user24544 Sep 5 '17 at 12:30
  • 2
    $\begingroup$ wrt 1: well, what did you read there (where?)? There are several things you can do - but what makes sense does depend on your optimization search space. I'm not sure a general answer is possible. 2.: you don't have to. But chances are that it is better as it has more training cases are available. (e.g. for single splits into training and test, the model may not be retrained. Technically the same but retraining is sometimes called set validation). For cross validation you'd end up not with a single model but with a set of surrogate models if you do not retrain. So then you need to decide how $\endgroup$ – cbeleites supports Monica Sep 5 '17 at 13:45
  • 2
    $\begingroup$ ... to use that set of models. (Averaging their predictions would be an ensemble model - which is different from the single model you evaluate by the cross validation) $\endgroup$ – cbeleites supports Monica Sep 5 '17 at 13:46
0
$\begingroup$

Although the answer has been already marked but I think it does not address the question correctly. Actually the answer lies in in the mention question itself. Just to express in a different way.

Two essential points here should have been explained:

  • Model checking: to determine the performance of models. In this case, a possible scenario is you have several learning algorithms and just want to select the best among them.
  • Model building: to build the model to use it for future prediction. In this case, you already know which algorithm you would use, you just need to train it as much as possible.

Cross validation is for model checking, because it allows to repeatedly train and test on a single set of data (if you had an unlimited amount of data, you would not need cross validation at all). But it is not for model building, since you want to exploit every single observation of the data you have in order to build a model to make prediction. In that case, instead of cross validation, you would need to train on the entire dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy