Yes, I know that after we fit (train) a predictive model on a training dataset, we need to test the fitted model on the testing-dataset. The motivation behind this procedure is clear: If our model performs great on the training data set, there is no guaranty that it will work well on the data that it did not "see" so far.
On practice it means the following. We have two (not more to keep it simple) candidate models (for example neural networks and decision trees). We fit each model on a training dataset. In other words we can say that we have two classes (or sets) of functions (corresponding to the two models) and by fit we select one function from each class.
Now we have two function (a neural network and a decision tree). We use each of these functions to check how good it is on a testing dataset. Than we choose the function that gives the best result on the testing set.
Now I try to make this procedure as simple as possible. We have four function and three data points. We randomly split the four functions into two classes (two function per class). From each class we choose one function that give the best prediction for the first data point. It means that we get two functions. From these two function we choose the function that performs the best on the second data point. Can we say that this is the best procedure to choose one function out of four such that it gives the best results for the third data point? Would it be not smarter to test all four functions on two first data points and then to choose the best one?
