Machine Learning - Training/Validation Sets

Question

I have a very general question that I can't seem to get a straight answer on.

Machine Learning - I understand how it works - you have your dataset for which you want to answer either a prediction or classification question.

For a prediction problem, you want to split your dataset into a training dataset first (usually ~80% of full dataset) and run your regression model on this data and look at a confusion matrix of predicted vs. actual response values to see how well your model predicts. Then, you fit the same model on the validation set (remaining 20% of the full dataset) and construct another confusion matrix to see if the % of correctly predicted responses is close to the % from the training dataset.

At this point, let's say you have equal %s. What is the next step? Do you then run your model on the full dataset now that you have confirmed it predicts well with different subsets of your data? Do you just use your training set? Any advice or explanations would be great and very appreciated!

Thank you!

Answer 1

Generally in a simple single-split training-validation setup, you fit the model on the training set and test it on the validation set without further fitting.

---

In testing, you should have some metric you're measuring - maybe MSE if you're doing continuous regression. The model you keep at the end is the one trained on the training set. The reason you measure performance on the validation set is to estimate overfitting; you usually want to know how much error you can expect when using the model on completely new data.

(Having quantified performance with a validation set, you could go back refit with the complete set, but you won't really know how much your final model overfits).

---

Depending on how demanding the fitting procedure is, you can attempt to get a more robust measure of overfitting using n-fold cross-validation. For example, in 5-fold CV, you would divide your dataset into 5 equal parts. For each combination of 4-parts-training, 1-part-validation, you fit on the 4-parts-training and test on the 1-part validation. That gives you 5 estimates of overfitting instead of just one. (The extreme case of this is leave-one-out CV, LOOCV, where each validation set has just one data point in it.)