In some lectures and tutorials I've seen, they suggest to split your data into three parts: training, validation and test. But it is not clear how the test dataset should be used, nor how this approach is better than cross-validation over the whole data set.
Let's say we have saved 20% of our data as a test set. Then we take the rest, split it into k folds and, using cross-validation, we find the model that makes the best prediction on unknown data from this dataset. Let's say the best model we have found gives us 75% accuracy.
Various tutorials and lots of questions on various Q&A websites say that now we can verify our model on a saved (test) dataset. But I still can't get how exactly is it done, nor what is the point of it.
Let's say we've got an accuracy of 70% on the test dataset. So what do we do next? Do we try another model, and then another, until we will get a high score on our test dataset? But in this case it really looks like we will just find the model that fits our limited (only 20%) test set. It doesn't mean that we will find the model that is best in general.
Moreover, how can we consider this score as a general evaluation of the model, if it is only calculated on a limited data set? If this score is low, maybe we were unlucky and selected "bad" test data.
On the other hand, if we use all the data we have and then choose the model using k-fold cross-validation, we will find the model that makes the best prediction on unknown data from the entire data set we have.