As usual, I have three sets of data: Train, Validation and Test. So I use train data for model selection, where I select the model which would perform best on validation data. After selecting the best model, if I use same Train data and Validation data to further tune or improve my model, and then test on Test set, am I doing something fundamentally wrong? Am I introducing any kind of bias or perhaps over-fitting? Is there something fundamental information that perhaps I am skipping?

  • $\begingroup$ What does it mean to select a best model but then improve upon it? $\endgroup$ – dsaxton Jan 15 '16 at 14:32
  • $\begingroup$ @dsaxton say I have n different models; I would choose one model, which upon training on train data performs best on validation data, usually one with least sum of squared errors. $\endgroup$ – Manu Jan 15 '16 at 14:48
  • $\begingroup$ Yes, that is a standard thing to do. What are you unsure about? $\endgroup$ – dsaxton Jan 15 '16 at 14:49
  • $\begingroup$ @dsaxton it's about whether we should train the selected model "again", but this time including larger set (train+validation data) to train, before deploying it on test data? $\endgroup$ – Manu Jan 15 '16 at 14:58
  • $\begingroup$ Unless I'm overlooking something, I don't see any issue with this as long as the test data are only seen once and you don't use the results to do further tuning. $\endgroup$ – dsaxton Jan 15 '16 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.