I've a time series from January 2010 to December 2017. I'm using 2010 to 2015 as the training set and I'm choosing the best model trained on the training set based on its performance in the validation/test set which is from January 2017 to December 2018 as measured by the validation/test MAPE.My objective is to have the validation/test MAPE as small as posdible (business requirement) Can the validation and test sets be the same in this case?

| cite | improve this question | | | | |

Normally, the validation set is used for deciding about the parameter values of the chosen model family, while the test set is used to evaluate the performance of the learned model. The test set should not be used during training at all, while validation set is used during training for choosing the suitable parameter values for the model. In other words, the test set is used only once to report the performance of the learned model, while validation set is used many times to evaluate different parameter settings.

| cite | improve this answer | | | | |
  • $\begingroup$ So, the validation and the test sets can never be same? $\endgroup$ – Shreyo Mallik Mar 17 '17 at 10:40
  • $\begingroup$ They can be the same, but it is not suggested since then you will probably have an overoptimistic evaluation of the learned model. $\endgroup$ – Hossein Mar 17 '17 at 11:14
  • $\begingroup$ Suppose I train 10 models based on the Training set. Could I choose the best model based on its performance in the test set? $\endgroup$ – Shreyo Mallik Mar 17 '17 at 11:15
  • 1
    $\begingroup$ It is better to choose the best based on the validation set, and finally report the performance based on the test set. $\endgroup$ – Hossein Mar 17 '17 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.