Yes, I know that after we fit (train) a predictive model on a training dataset, we need to test the fitted model on the testing-dataset. The motivation behind this procedure is clear: If our model performs great on the training data set, there is no guaranty that it will work well on the data that it did not "see" so far.

On practice it means the following. We have two (not more to keep it simple) candidate models (for example neural networks and decision trees). We fit each model on a training dataset. In other words we can say that we have two classes (or sets) of functions (corresponding to the two models) and by fit we select one function from each class.

Now we have two function (a neural network and a decision tree). We use each of these functions to check how good it is on a testing dataset. Than we choose the function that gives the best result on the testing set.

Now I try to make this procedure as simple as possible. We have four function and three data points. We randomly split the four functions into two classes (two function per class). From each class we choose one function that give the best prediction for the first data point. It means that we get two functions. From these two function we choose the function that performs the best on the second data point. Can we say that this is the best procedure to choose one function out of four such that it gives the best results for the third data point? Would it be not smarter to test all four functions on two first data points and then to choose the best one?


1 Answer 1


I think what you are saying is that the usual procedure of training different models on a training set and then using a validation set to tune hyperparameters is an inefficient use of data. I agree with that. However, making more efficient use of the data is not as simple as you suggest. In your toy example, you seem to be using empirical risk minimization which is a bad idea since it will overfit in general. Thus your second procedure will not necessarily generalize well. The first procedure that you mention is a strawman since a practitioner wouldn't normally make a random split of the four functions into two classes. The split should be made with some care.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.