For model building I typically apply the following process (which I am simplifying somewhat here for brevity):
- Split the data into test and training
- Use cross-validation on my training data to find the best model parameters relative to some performance metric (accuracy, ROC etc) and pick this as my best model
- Evaluate my model on the test data
So far so good.
One problem that arises is that my test data is often valuable to me. Suppose I am predicting the value of a stock. On the one hand, I want my test data to be the most recent data so that I can see how the model performs on recent stock prices. On the other hand, I want to fit my model on my latest stock price data because that is the most relevant data for predicting tomorrows stock price.
But After applying 3. above, I have a model that is not using the test data to fit the model.
My question is, is it legitimate (or standard practice) to refit the model with the final tuning parameters to the entire data set as a final fourth step. For example, suppose in 2. my best parameter for a random forest model is mtry = 5. So at the end of 3. I refit the entire model with mtry = 5. Note, using seeds in R I am able to ensure the best model I picked in 2. will be identical in form to the model I fit in this final step i.e. I do not want to refit the final model with 5 new random features - I want to use the same features as the model I picked in step 2.
Thoughts please. Is this good / bad? Does it violate some fundamental principal in machine learning approach?