# How do to model selection and training?

I have read other posts here. But i am still confused. As far as i understood there are two procedures:

1) Split data into train, validation and test set . In this case my concern is that once we have chosen the model, do we combine the Validation and Train set to train the selected model and then report the error on test set or we don't combine the Validation and Train set?

2) Cross Validation : In this case once we have the model, do we train it on the whole Data set or we set aside some as test set ? What do we report as the final performance measure of the model ?

Please clarify what is done, I have read over internet but can't find any conclusive answer to it.

1) Generally, yes, you can add the train and validation sets for final training because you'd be throwing out the validation set otherwise (actually not entirely since you'd used information from it to select a model, but pretty much).

2) From the model trained on the entire dataset, expect performance equal to the average performance over the folds of CV. It feels like kind of a leap of faith because you don't actually see the full model performance on specific cases, I know, but it's accepted in practice.

To report on the quality of your model in machine learning, you always split the data into two parts, the training dataset (typically 2/3 of the data) and the test dataset (typically 1/3 of the data). With the training dataset you fit the model and with the test dataset you evaluate your model. You report the prediction quality of your test dataset.

Reading the Wikipedia page on "Cross-validation (statistics)", cross validation basically randomly draws a sample of, say 2/3, the data again and again as training dataset and always evaluates it with the remaining test dataset. You report the average prediction quality of test datasets over all iterations. Therewith multi-iteration cross validations makes a more effective use of the data than a single-iteration split into training and test dataset.

This said, when applying your machine learning model for observations where you don't know the classification or dependent variable, you of course train your model beforehand on all data with known classification or dependent variable you have.

The whole point of model validation is to avoid selecting a model that overfits or underfits to your data, by assessing predictive power of your model. A model that overfits to the data, is too complex and captures both the signal and the noise in the data. A model that underfits to the data, is not complex enough, and struggles to capture the signal.

There are several methods to carry out validation:

1 : One approach would be to split your data set into a training set and a test (roughly 70/30 split). Then you can assess how well your model generalises to the unseen test data. For example, with a simple linear model, we would make predictions on the test data, then use mean squared error (MSE) as a measure of predictive performance.

2: As second approach would be to use k fold cv (cross validation). In this approach the data set is split into k disjoint subsets called folds. Then for each of the subsets the following process is carried out:

2.1 : Train your model on a set of data which excludes the current fold.

2.2 : Now make predictions on the unseen fold .

2.3 : Calculate a measure of predictive performance such as MSE.

$\mbox{MSE} = \frac{\sum_{t = 1}^{N}(\hat{y_{t}} - y_{t})^2 }{N}$

This leaves us with $k$ measures of predictive performance. Now we simply take the average of these measures to give an overall measure of predictive performance, the cv score. A low cv score would indicate good predictive performance. A high score would indicate that we have overfitted or underfitted, i.e our model does not generalise well to unseen data.

N.B : The folds should be of similar size, and have similar statistical properties. This can be accomplished by randomly allocating data points to a fold.

1. Leave one out cv (LOOCV) is a special case of the above method when $k = 1$. Each fold simply consists of a single data point. This can be computationally expensive, as we have to fit the model $N$ times, where $N$ is the size of your data set.

2. Out of the bag validation (OOB) is another approach. In this approach we take $B$ random subsets from our data set. These subsets now form the test data sets, much like the folds in 2. The process now proceeds much along the same lines as in 2.

One measure that should not be relied upon for validation is $R^{2}$. Having a high $R^{2}$ does not mean that your model has high predictive performance. This measure was calculated using data that trained your model. Only data that the model has not "seen", can provide a measure of predictive performance.

I should note, that if you can compute the AIC for a model, this can also be used as a measure of predictive performance. The reason being since AIC is asymptotically equivalent to a score computed by LOOCV.

Sometimes validation involves train, validation and test datasets. I am sure this terminology and approach is due to SAS. It is something that I personally don't bother with and just use train and test data sets.

Hope this helps

• All of this i know, you have not answered my question. Do we finally train on the whole data set in case of cv ? Aug 27 '17 at 13:42
• I would say that once you have validated your model, you would use all available data to estimate model parameters. Aug 27 '17 at 17:50