As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to contaminate my testing and validation sets and run the risk of overfitting.
Let's say I have split my data into a training, testing and validation data set. I would like to try several algorithms - e.g. logistic regression, RF, SVM - and pick the best of them.
- May I train and test all three of the models, or only one of them?
- Can I use the training set alone (i.e. in cross validation) to internally test multiple models?
- Given I have a validation set, what may I do after having used up my testing set? Tweak parameters of the models? How many times?
- If I combine several models into one (ensemble learning), in which step would I do that?
- In your opinion when looking at my question - is there something I have fundamentally misunderstood about the training/testing approach?