4
$\begingroup$

I am going through the excellent book "Introduction to Machine Learning with Python," and reading about cross-validation. I can understand how it makes a more efficient use of the data than a typical train-test split, but the book also contains the caveat:

It is important to keep in mind that cross-validation is not a way to build a model that can be applied to new data. Cross-validation does not return a model... multiple models are built internally, but the purpose of cross-validation is only to evaluate how well a given algorithm will generalize when trained on a specific dataset.

So if cross-validation doesn't produce a model, does that mean that after performing cross validation, I need to build a model in the typical method using a train/test set? If so that would imply that my cross-validation scores would typically be higher than my final model's scores, since cross-validation makes a more efficient use of the data.

Or is it held that after cross-validation, I can simply train my model against all data without any further test set? That would mean that I've never tested my actual model, so it sounds wrong, but perhaps cross-validation is a valid test since it uses every sample in both training and test? If so it implies that my cross-validation scores would typically be lower than my final model's scores, since only my final model would be trained against all samples.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Or is it held that after cross-validation, I can simply train my model against all data without any further test set?

Yes - the cross validation is a (more efficient) replacement for that test set.

That would mean that I've never tested my actual model, so it sounds wrong, but perhaps cross-validation is a valid test since it uses every sample in both training and test?

CV treats its training sets as a good approximation to the whole data set (as do other types of resampling validation such as out-of-bootstrap etc.), so it is approximately right.

There are numerous studies on the error you make with different validation schemes that consider the total = systematic + random error (bias + variance). Turns out, with sample sizes (< a few 1000 independent cases) it is better than the alternative of train-test-split where - as you say - you have the advantage of being unbiased but pay for this with much higher variance.

If so it implies that my cross-validation scores would typically be lower than my final model's scores, since only my final model would be trained against all samples.

Yes - cross validation will have a slight pessimistic bias if done correctly (depending on the slope of the learning curve between the CV train sample size and the total sample size). You trade that for less variance (depending on the test sample size for the train-test split and the total sample size for CV).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Upvoted for the thoughtful answer. I am still wrapping my head around these concepts and going through the book. My goal is to accept an answer once I understand everything better, since the two answers are somewhat different in emphasis. $\endgroup$
    – Stephen
    Commented May 14, 2017 at 18:52
  • $\begingroup$ One question I do have about your answer: How does the train-test split have the advantage of being unbiased? I get that CV has the slight bias of having a smaller training size than the total sample size, but the train-test split would have this too. $\endgroup$
    – Stephen
    Commented May 14, 2017 at 19:04
  • $\begingroup$ @Stephen, the train-test split claims that train is the training set, and there is no model build on all data. The CV estimate is extrapolated: it is assumed to be a good estimate of the generalization performance of a model trained on all data. (BTW, a similar extrapolation exists for train-test splits, which I've met under the name of set validation - and yes, I think there's a whole lot of confusion and differing vocabulary around with these topics) $\endgroup$
    – cbeleites
    Commented May 17, 2017 at 8:32
4
$\begingroup$

The phrase from the book doesn't sound right to me, but maybe I don't have the whole context.

Let's assume that we split your whole data set into two: train and test (for example 70%, 30%).

If we apply a cross validation scheme to the training set (not touching test set) by splitting it into several sections (folds), and run the machine learning algorithm with the intention to find the values of hyperparameters that produce the lowest training error, we of course end up with a model (meaning that we also find the values of the parameters of the model).

If we think for a moment why we are doing cross validation, we see that the reason is to reduce the number of freely adjusting parameters. That's why the number of hyperparameters are almost always lower than the number of model parameters. We do this because the number of free parameters increases the bias (very optimistic) in estimation of the expected training error.

Having said that, the cross validation still has freely varying hyperparameters. Therefore the bias is reduced but not completely eliminated. For this reason when you actually run the model emitted by the cross validation on the test set (30% part) the performance will still be worse "on the average" compared to the training.

That's why it is a good idea to still keep a test set aside even if you are building your model with cross validation. The error of the model on the test set is the most honest estimate of expected error.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Upvoted for the thoughtful answer. I am still wrapping my head around these concepts and going through the book. My goal is to accept an answer once I understand everything better, since the two answers are somewhat different in emphasis. $\endgroup$
    – Stephen
    Commented May 14, 2017 at 18:52
  • $\begingroup$ I do have one question about your answer: You say that cross-validation (or perhaps "grid search with cross validation") is done to reduce the number of freely adjusting parameters. My understanding so far is that this is done to choose such parameters. This process will produce bias because of the likely overfitting of these hyperparameters. That part I understand, but is there something more you're referring to here other than the evaluation of many parameters to choose the best one? Perhaps the decision not to use some parameters at all? $\endgroup$
    – Stephen
    Commented May 14, 2017 at 19:12
  • $\begingroup$ Everytime you adjust and test a parameter on the same data set you are introducing a bias. Model parameters are set and evaluated on training set hence create a bias. Hyperparameters are set and evaluated on the validation set hence create a bias but lower. I don't know what you mean by not using parameters. Without parameters there's nothing to train. $\endgroup$
    – Cagdas Ozgenc
    Commented May 14, 2017 at 19:37
  • $\begingroup$ Sorry, let me rephrase my question to be clear: I don't understand what you mean by "reduce the number of freely adjusting parameters." I don't see why cross-validation "reduces" parameters, I thought it just "chooses" them. $\endgroup$
    – Stephen
    Commented May 16, 2017 at 1:57
  • $\begingroup$ It doesn't reduce in the sense of removing them, but hyperparameters limit the free movement of the parameters of the model during the optimization. For example in ridge penalty, the total squared weights are bounded by lambda hyperparameter. $\endgroup$
    – Cagdas Ozgenc
    Commented May 16, 2017 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.