Can this approach be used for machine learning using train-test split?

Question

So let's say I have a dataset with 1000 samples, 20 cols. Regression problem.

I use train-test split, say 80-20%

I create a Model, lets say Random Forest. I use gridsearchCV to find the best model that gives me highest R^2 for the same. Now, imagine this is a kaggle competition, that gives us a result or a score on a different test set when we submit our predictions.

Now, IF I use the same parameters that gridsearchCV gave, but train the model on the entire training data, and use that to test the unseen data, should I get better results? Or should it be the same?

My assumption is that, by feeding more data, and having the same parameters, I'll only generalise the model more, which should be good, and will lead to a minute increase or decrease in the kaggle score.

Please do give your opinions about it. Thanks!

Answer 1

You cannot really predict the outcome. Ideally when you train your models on a subsample of your data (for ex. doing "hyperparameter tuning") and then train it on the whole data you should get similar results... assuming the data is similar between train/validation, the validation data is a smaller sample of the total, the model is not very sensitive to data quantity, etc.

There are many ways in which this could go wrong and not work as expected, for ex. maybe the data in your training sample was easier to predict than the validation data (it doesn't have to be by much), or vice versa. In these cases you will probably get different results once you include all the data (over- or under-fitting).

Another likely culprit is the model used, for ex. if you are using an iterative model such as RF, where one of the parameters is the number of trees, then you might get different results when adding the validation data, as this would increase the number of trees required to achieve the same performance.