1
$\begingroup$

After dividing dataset to train, validation and test set I use validation set to determine the best hyperparameters. After that I need to evaluate performance on test set. For this evaluation do I use merged train and validation set or just train set for training?

And how is this done if dataset is divided into train, test set and use crossvalidation on train set to find the best parameters? Here I probably use all train set for training final model and then test on test set?

$\endgroup$
0
$\begingroup$

So basically, when evaluating a model you use the test set. That means in case of crossvalidation there is no confusion. Always remember that the training data is only used for training purposes (unless you want to check for overfitting, and even for that, there are better ways).

I hope this helped.

$\endgroup$
  • $\begingroup$ Ok. But when evaluating on test set, which data should I use for training the final model? Just training data or merged together training + validation data. $\endgroup$ – tskr May 2 '18 at 13:19
  • $\begingroup$ Once the validation phase is done, you don't train one more time. The process goes as follow: optimize the hyperparameters by training and validating, You then evaluate the 'best model' using the test set. If the results are not satisfying, you start from the top with different features/model etc.. $\endgroup$ – Amani May 2 '18 at 13:30
  • $\begingroup$ @tskr and Amani, the model may be reestimated on training+validation set without changing the tuning parameters and then tested on the test set. That allows for better accounting for the sample size as the real sample you have is the sum of training, validation and testing. $\endgroup$ – Richard Hardy May 2 '18 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.