The classical way in machine learning is to split the dataset into three parts: train, validation and test. The train dataset is used to update the weights, the validation dataset is used during to evaluate the model during training and e.g. perform early stopping and the test set is used once at the very end to get a final score.

When I use cross validation, I still have a separate test set, but I split the remaining samples into k-folds and train/validate with them k times.

According to this answer, I use the whole dataset (that is not reserved for test) after cross validation to train my final model. Does that mean I don't validate here while training the final model?

To clarify: I'm talking about how to train the final model after I did cross validation to select an algorithm and tune the parameters.

  • $\begingroup$ The link you provide explains it well. Use CV to find your model. Once selected you train it on the entire dataset, so there is no further validation. To report generalisation error you use what you found in the CV. After deploying the model in the real world you might get some more metrics to assess its performance, but I think that's a different topic. $\endgroup$
    – N Blake
    Apr 15 at 10:39

You don't always have to validate, since a lot of times various CV techniques are employed for training and testing. See, for example, Kohavi

$k$-fold cross-validation. One of the most efficient methods for building the confusion matrix C is to use $k$-fold cross-validation. Let $\mathcal{D}$ represent all of the objects in a dataset. In $k$-fold cross-validation, we split $\mathcal{D}$ into $k$ mutually exclusive partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_k$ and the classifier is trained and tested $k$ times. In 10-fold cross-validation, we set $k=10$. Stratified 10-fold cross-validation is then carried out using ten repartitionings for each classification run. During a repartitioning, the order of objects is first shuffled and then objects are assigned to the 10 ordered partitions (folds), while ensuring that at least 1 objects from each class is in each partition. Using the 10 partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ the classifier is first trained with objects in partitions $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ and objects in partition $\mathcal{D}$$_1$ are then used for testing (class prediction). Next, we train with partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_3$,$\ldots$,$\mathcal{D}$$_{10}$ and test objects in partition $\mathcal{D}$$_2$. This is repeated until training with partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{9}$ and test objects in partition $\mathcal{D}$$_{10}$ is completed. This scheme is repeated 10 times, each time randomly assigning all objects into 10 partitions and then performing training and testing on objects in partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ sequentially.

Leave-one-out cross-validation. Another method for constructing the confusion matrix involves leave-one-out cross-validation or LOOCV, where $k=1$. In LOOCV, class prediction is made on each test objects when is it held out from training. LOOCV is known to be a pessimistically unbiased estimate of the true error rate, but like other approaches does not guarantee the same accuracy for future objects.

0.632 Bootstrap. This technique determines classifier accuracy using cross-validation based on the 0.632 bootstrapping method. For the $b$th iteration $(b=1,2,\ldots, B)$, objects are randomly selected with replacement and called sample $\mathcal{D}^b$. The probability that an object is not selected during bootstrapping is $(1-1/n)^n=\exp(-1)=0.368$, and we call this sample of objects $\mathcal{D}^0$. Next, the classifier is trained using $\mathcal{D}^b$. The classification accuracy of objects in $\mathcal{D}^b$ is known as the apparent accuracy and is denoted as $Acc_b$. The classification accuracy of objects in $\mathcal{D}^0$ based on training with $\mathcal{D}^b$ is denoted as $Acc_0$. The steps necessary for CVB are listed below:

  1. Select $n$ objects with replacement, and call this the ``bootstrap'' sample $\mathcal{D}^b$. Call the set of objects not selected sample $\mathcal{D}^0$.

  2. Train the classifier using objects in $\mathcal{D}^b$.

  3. Obtain $Acc_0$, by predicting class membership of objects in $\mathcal{D}^0$.

  4. Obtain $Acc_b$, by predicting class membership of objects in $\mathcal{D}^b$.

  5. The 0.632 cross-validation accuracy is determined with the relationship \begin{equation} Acc_{0.632} = 0.368 Acc_b + 0.632 Acc_0. \end{equation}

The above steps are carried out, for example, $B=40$ times and the average of $Acc_{0.632}$ over the $B$ iterations is used as the final estimate of accuracy.

Holdout method. The holdout method for constructing the confusion matrix partitions the dataset $\mathcal{D}$ so that, for example, two-thirds of the objects are used for training and the remaining 1/3 are used for testing. Random subsampling is often repeated $k$ times, in which objects are randomly selected followed by accuracy determination. A drawback of the holdout method is that it makes inefficient use of $\mathcal{D}$, since, for example, a third of the objects are not used for training the classifier.

In the image below for training and testing error in 10 folds of a neural network, you can see that in folds 3,4,5,6,9 and 10 that the training error (red) continued to decrease over the epochs while the testing error has increased. When error for testing and training start to diverge, you know the model is over-fitting, since test error is breaking down.

enter image description here

  • 1
    $\begingroup$ That doesn't fully answer my question. I know the principles of cross validation or bootstrap in general, I just wondered how to create the final model that I would use e.g. in a product then. Do I just train on 100% of the training data and don't validate during training? $\endgroup$ Apr 15 at 9:44
  • 1
    $\begingroup$ cv is just for parameter adjustment and to know what estimator and so on are good. If you are finished you build your pipeline on all your data. It can be still training data though. imagine you have 40 million rows in train and 5 mill in test. Then it would be suffiient if everything was fine. to just use train data, as fitting and deploying on all 45 would take for example 1-2 days longer and you are under time pressure in deploying and delivering. Than train data would be sufficient. to sum up: validation is done, you know your framework, deploy it. $\endgroup$ Apr 15 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.