When does it makes sense to use Cross Validation?

Question

My understanding is that cross validation is about using different chunks of the training data to train the model and average out the error estimation so that the variance is less. For example, in Leave One Out Cross Validation, we use 1 row for test n-1 rows for training. This is slow and we prefer 10 fold cross validation where the data is divided into 10 chunks where each chunk serves as validation data while rest will be used for training.

Where I don't get the concept is when should I be using Cross Validation ? I am working on Kaggle's Bike Sharing Demand data and I have used Linear regression, Random Forest as well as Generalized Boost models. Out of those, so far Random Forest gives me lowest Root Mean Squared Log Error.

Now, in this scenario, will it make sense to do 10 fold cross validation for each model ?

Answer 1

Here are 2 different scenarios where cross-validation can be used.

1) You want to approximate your model's generalization error (how well it will do on inputs it hasn't seen before). Cross-validation can tell you that because it trains on one set of data, and tests on the other set of data. The error on the test set is representative of the generalization error.

This is important because it would be trivial to get 100% on your training set - just create an n-way if-statement for your n training samples. That is called overfitting.

2) You want to choose 'hyperparameters'. In gradient descent, for example, the learning rate is a hyperparameter. You would do cross-validation for different settings of the learning rate, i.e. 0.01, 0.02, ... and choose the one that gave you the lowest error. If you do happen to do this, the 'test' error you got won't be representative of your generalization error as in #1 - why? Because you were essentially 'training' your learning rate based on different train/test splits of your data (so in a way it was all training data). And thus another set of data would have to be used outside of that to determine the generalization error.

Answer 2

In addition to @justin_credible's points:

 When does it makes sense to use Cross Validation?

• Whenever you cannot afford the independent test set you'd really want to have.
• Iterated $k$-fold CV or out-of-bootstrap: in order to measure the stability of the predictions wrt. to slight changes in the training set.
  (Note that this may calculate more surrogate models than leave-one-out CV)