ML / train-test-validate: What is allowed when?

Question

As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to contaminate my testing and validation sets and run the risk of overfitting.

Let's say I have split my data into a training, testing and validation data set. I would like to try several algorithms - e.g. logistic regression, RF, SVM - and pick the best of them.

• May I train and test all three of the models, or only one of them?
• Can I use the training set alone (i.e. in cross validation) to internally test multiple models?
• Given I have a validation set, what may I do after having used up my testing set? Tweak parameters of the models? How many times?
• If I combine several models into one (ensemble learning), in which step would I do that?
• In your opinion when looking at my question - is there something I have fundamentally misunderstood about the training/testing approach?

Answer 1

Let's begin first with the train and test set. The correct way to go is to train several models using your train set and compute the error in the test set. The best model is the one with the lowest test error.

However, almost every model has parameters to tune. Note that you can't use the test set to do that or you will be overfitting your model to the test data and it would be like "cheating". You have two options: either you have a validation set, and you tune your parameters so they have the lowest error in the validation set or you don't and use cross validation with your train set. Note that you can always split randomly some data from the train set in order to make a validation set, but maybe you don't want to do that because there is not many examples to begin with.

In any case, once the best parameters are selected you compute again the error in the test set. Given some assumptions (the distribution of the test set is similar to train and validation and so on) this is a good estimate of your generalization.error because you didn't use the test set to train the model nor to tune the parameters, so if you were to use this model on new data (again given the same assumptions) you would expect to obtain an error similar to that one. Note here that you cant measure this, because usually for this new data the real.outcome is not available and you can only hope. This would be the production phase and your model is ready for business!