8
$\begingroup$

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on a separate, validation set.

You do all your heuristics on the training set and when you are happy with the selected parameters, you obtain a model and test it on the testing set. Sometimes you will fine-tune your model, according to the testing set results, but you keep these tunings to the minimum.

Now you are ready to publish your model and to validate it using an external, validation, set of data. Would you re-train the model (re-calibrate the parameters) using the combination of the training and the testing set, or would you stick to what you have?

$\endgroup$

1 Answer 1

9
$\begingroup$

I would retrain the model using the training and test partitions; there doesn't seem much to lose from doing so, and the final model will probably perform slightly better than the pessimistic (as it is based on a smaller training set) performance estimate.

If the dataset is small, a better solution would be to combine the training and test sets and use bootstrap replication. You can use the out-of-bag performance estimator for whatever fine tuning you need to do, and you can just use the same bootstrap committee (i.e. bagging) for the validation set predictions.

For small datasets bagging is very useful because if you use a single test/training split, the test set performance is highly variable die to the small size of the test set, so it is an unreliable estimator of true performance. The out-of-bag estimator generally has a lower variance and is a better guide to performance. Likewise the predictor itself is less variable with bagging that if trained on a single training set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.