You don't always have to validate, since a lot of times various CV techniques are employed for training and testing. See, for example, Kohavi
$k$-fold cross-validation. One of the most efficient methods for building the confusion matrix C is to use $k$-fold cross-validation. Let $\mathcal{D}$ represent all of the objects in a dataset. In $k$-fold cross-validation, we split $\mathcal{D}$ into $k$ mutually exclusive partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_k$ and the classifier is trained and tested $k$ times. In 10-fold cross-validation, we set $k=10$. Stratified 10-fold cross-validation is then carried out using ten repartitionings for each classification run. During a repartitioning, the order of objects is first shuffled and then objects are assigned to the 10 ordered partitions (folds), while ensuring that at least 1 objects from each class is in each partition. Using the 10 partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ the classifier is first trained with objects in partitions $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ and objects in partition $\mathcal{D}$$_1$ are then used for testing (class prediction). Next, we train with partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_3$,$\ldots$,$\mathcal{D}$$_{10}$ and test objects in partition $\mathcal{D}$$_2$. This is repeated until training with partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{9}$ and test objects in partition $\mathcal{D}$$_{10}$ is completed. This scheme is repeated 10 times, each time randomly assigning all objects into 10 partitions and then performing training and testing on objects in partitions $\mathcal{D}$$_1$, $\mathcal{D}$$_2$,$\ldots$,$\mathcal{D}$$_{10}$ sequentially.
Leave-one-out cross-validation. Another method for constructing the confusion matrix involves leave-one-out cross-validation or LOOCV, where $k=1$. In LOOCV, class prediction is made on each test objects when is it held out from training. LOOCV is known to be a pessimistically unbiased estimate of the true error rate, but like other approaches does not guarantee the same accuracy for future objects.
0.632 Bootstrap. This technique determines classifier accuracy using cross-validation based on the 0.632 bootstrapping method. For the $b$th iteration $(b=1,2,\ldots, B)$, objects are randomly selected with replacement and called sample $\mathcal{D}^b$. The probability that an object is not selected during bootstrapping is $(1-1/n)^n=\exp(-1)=0.368$, and we call this sample of objects $\mathcal{D}^0$. Next, the classifier is trained using $\mathcal{D}^b$. The classification accuracy of objects in $\mathcal{D}^b$ is known as the apparent accuracy and is denoted as $Acc_b$. The classification accuracy of objects in $\mathcal{D}^0$ based on training with $\mathcal{D}^b$ is denoted as $Acc_0$. The steps necessary for CVB are listed below:
Select $n$ objects with replacement, and call this the ``bootstrap''
sample $\mathcal{D}^b$. Call the set of objects not selected sample
$\mathcal{D}^0$.
Train the classifier using objects in $\mathcal{D}^b$.
Obtain $Acc_0$, by predicting class membership of objects in
$\mathcal{D}^0$.
Obtain $Acc_b$, by predicting class membership of objects in
$\mathcal{D}^b$.
The 0.632 cross-validation accuracy is determined with the
relationship \begin{equation} Acc_{0.632} = 0.368 Acc_b + 0.632
Acc_0. \end{equation}
The above steps are carried out, for example, $B=40$ times and the average of $Acc_{0.632}$ over the $B$ iterations is used as the final estimate of accuracy.
Holdout method. The holdout method for constructing the confusion matrix partitions the dataset $\mathcal{D}$ so that, for example, two-thirds of the objects are used for training and the remaining 1/3 are used for testing. Random subsampling is often repeated $k$ times, in which objects are randomly selected followed by accuracy determination. A drawback of the holdout method is that it makes inefficient use of $\mathcal{D}$, since, for example, a third of the objects are not used for training the classifier.
In the image below for training and testing error in 10 folds of a neural network, you can see that in folds 3,4,5,6,9 and 10 that the training error (red) continued to decrease over the epochs while the testing error has increased. When error for testing and training start to diverge, you know the model is over-fitting, since test error is breaking down.
