3
$\begingroup$

I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a protein bound to a ligand is used to predict the binding affinity of the complex. I have a training set of ~3600 examples and a test set of 209 examples. The test set is chosen to be representative and diverse (more details on how exactly this works can be found in [1]). The test and training set are disjoint. The goal of the test set is to evaluate the performance of the model on a diverse set of unseen examples. I am testing out many different regression models (MARS, kNN regression, support vector regression and random forest regression). I used a grid search with 10-fold CV to tune hyperparameters on the training set, and with the optimal hyperparameters, I trained the model on the same training set. Following this, I tested and compared model performance on the testing set.

How good or bad of a practice is it to tune hyperparameters and train the model on the same set? What is the consequence of doing this?

References: [1] Ashtawy HM, Mahapatra NR. A Comparative Assessment of Predictive Accuracies of Conventional and Machine Learning Scoring Functions for Protein-Ligand Binding Affinity Prediction. IEEE/ACM Trans Comput Biol Bioinform. 2015 Mar-Apr;12(2):335-47. doi: 10.1109/TCBB.2014.2351824. PMID: 26357221.

$\endgroup$
2
  • $\begingroup$ This is likely to lead to a biased performance estimate and over-fitting of the test data. Mrs Marsupial and I wrote a paper on this sort of thing which you may find helpful ( jmlr.org/papers/volume11/cawley10a/cawley10a.pdf ). Note we were largely concerned with over-fitting a cross-validation based model selection criteria - using the test data for the model selection criterion is likely to result in a far greater performance bias. $\endgroup$ Apr 4, 2023 at 16:12
  • $\begingroup$ Whatever data you use to tune hyperparameters becomes part of the training set by definition -- your model is built using information in this set. $\endgroup$ Apr 4, 2023 at 16:37

2 Answers 2

17
$\begingroup$

It's completely inappropriate to pick hyperparameters from the test set, if it's meant to be a test set, because it makes the results on the test set unreliable. I.e. even if the test set is perfectly representative of what the model would be used on in real-life, there's no longer any reason to believe the test set performance would also be seen when the model is used in practice. A set of data used in this way is not a test set (terms used for it would e.g. be "validation set").

How badly the performance evaluation is wrong depends heavily on the details of each case. With a enormously large test set to which one could not possibly overfit, it may be less of a problem, but 209 examples is a really small number, so the potential for issues is large. The other thing is how heavily one can tune the models. E.g. just tuning the L2-penalty of a ridge-regression is probably less dangerous compared with tuning many parameters (e.g. regularization parameters, data pre-processing choices, feature sets to include etc.). At some point the number of hyper-parameters could be so large, that one could almost achieve a perfect fit to the set used for hyperparameter tuning.

It is much more typical to tune hyperparameters and select models based on cross-validation and/or a validation set. Comparing models on the test set could make sense, if your study is primarily about comparing the models. If the purpose is to then select one model and say what it's performance is, you end up overestimating it's performance by selecting the best model out of several models. An unbiased estimation of its performance could then come from a new test set.

$\endgroup$
4
  • 3
    $\begingroup$ Test set and validation set are not consistently used, I have seen both used to refer to the holdout. en.wikipedia.org/wiki/… $\endgroup$
    – Brady Gilg
    Apr 3, 2023 at 21:00
  • $\begingroup$ I apologize for the confusion. The title of my question has been corrected. I am training my model on the same training set that I tuned the hyperparameters on. Is this a bad practice? $\endgroup$ Apr 4, 2023 at 10:12
  • 1
    $\begingroup$ If you use a good validation set-up (see e.g. this discussion of what constitutes a good validation), which might be a suitable form of cross-validation (but e.g. in time series it might be suitable past-vs.-future splits), then that's common practice. You'd want to evaluate performance on separate data ideally even from a totally different data source ("external validation" as opposed to a train-test-split or cross-validation, which are forms of "internal validation" - i.e. internal to the original data source). $\endgroup$
    – Björn
    Apr 4, 2023 at 13:38
  • $\begingroup$ Note it is possible to over-fit cross-validation based model selection criteria as well if you have many models to compare, especially if you also tune hyper-parameters using the same criterion. You may get away with it if there are few hyper-parameters and all you want to do is choose the best model - the cross-validation performance will give an optimistically biased performance estimate. $\endgroup$ Apr 4, 2023 at 16:15
2
$\begingroup$

Tuning hyperparameters to training data is, for the most part, the same as fitting any other parameters to the training data. It's what the training data is for, but care needs to be taken to not overfit the data.

Often a model's hyperparameters directly affect the model's complexity so just taking the hyperparameters that lead to the best results on the training data will likely lead to overfitting. In theory, performing 10-fold cross-validation for the hyperparameter search should alleviate this issue, but depending on exactly how you're doing the cross-validation and measuring the performance you could get better or worse results.

  • The training/validation sets should be chosen first. You want these to be the same for each set of hyperparameters in order to reduce the randomness of choosing those sets as a variable. Otherwise, the chance that for a given set of hyperparameters the cross-validation sets are easier to get good results on for whatever reason may affect the final results.
  • Make sure the model is being retrained for each set of hyperparameters and each training/validation set. You should not train on the entire training set and then evaluate on a random validation set that is a subset of the training set.
  • For a given set of hyperparameters after you evaluate the performance for each training/validation set make sure you average them in some way, don't just take the min error. This again has to do with minimizing the effect of favorable random splits for certain parameters.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.