Deploying a machine learning algorithm, should I include the validation set? We have developed, trained, and tuned a random forest algorithm for a certain task. We followed the classic split of training/test/validation and we are satisfied with the performance of the algorithm at the moment.
My question is, for production/deployment, should we fit the algorithm on all the data (including the validation set)? We have sufficient self-control that no one will touch that fitted object for any thing other than the production deployment.
Please provide pros and cons for your answers (with references if possible).   
 A: How do you define validation?
Normally it is train, validation=dev=tuning, test; I will proceed according to this.
You have trained on train+val and tested on test.
I think you are asking whether to further train on train+validation+test.  If you train on train+val+test you need to use nested CV or bootstrap to obtain a performance estimate. The performance estimate you got from the test set for a model trained on train+val does not apply to this model trained on train+val+dev.  For example, you might have set a HP that gives best results for the tuning set when training on the train set, but it is too restrictive for training on the full set.  Most HP prevent overfitting—-so does more data—-so you likely could have imposed a less severe penalty to get best performance of a model fit on all the data.  I think usually this would be the case, so your model would actually slightly underfit the full set, which is good because you won’t be optimistic with your error estimate.  However not really sure. Your train+val set might be a very weird sample and have caused you to actually select a HP that leads to overfitting even on the full data.
I would not feel comfortable reporting a performance estimate for a model trained on the full dataset if the estimate was made for a model trained on train+validation and tested on a test set.  Better to just use the model you have (traInès on train+validation), even though you throw out some data, or use nested CV or bootstrap to correctly obtain performance estimate for model on full data.  Better to be wrong but know it than be wrong but not know it, even if you don’t make full use of your data.
