1
$\begingroup$

The shorter and more general version of this question: If tuning a model via cross-validation (within training set) produces worse results on the test set than my previous default/baseline model, do I stick with the tuned model (to avoid over-fitting on the training data) or do I go back to the baseline (even though it seems like I'm overfitting by doing so)?

Long more detailed scenario: Let's say I have an 80/20 train/test split set. Then I build a model with default model hyperparameters and obtain an F1 score of 0.35.

Then I use cross-validation on the train set to identify the best hyperparameters and build a new model on all the training data using those hyperparameters found optimal per the cross-validation. However, when I evaluate this "optimal" tuned model on the test set I get an F1 score of 0.23.

In such cases, should I stick to the default hyperparameters that had produced the higher F1 score on the test set or stick with the tuned model since it was tuned using cross-validation?

In case such variation in numbers is unlikely - I guess then I'm wondering whether there may be some other factor at play, such as too small a dataset (e.g. around total 1000 datapoints total with fewer than 200 of them being in the class of interest) or imbalanced fold partitioning.

$\endgroup$
  • $\begingroup$ Where did the defaults come from? Are they just the defaults supplied by the model-fitting software for the type of model in question? $\endgroup$ – Kodiologist Aug 12 '16 at 6:23
  • $\begingroup$ In this case, yes. However, I'm also interested in the more general case where one has arrived at some new 'go-to' model that does well on the test set (say after successfully tuning for feature selection), but then attempts to tune some new aspect (say a cutoff probability using cross-validation) and the evaluation scores on the test set get worse. $\endgroup$ – blu Aug 12 '16 at 16:15
  • $\begingroup$ By "cross-validation (within training set)" you mean that you're doing something like nested CV: an outer CV which takes each of its training sets and calls an inner CV on that training set, with data prep, etc, being done within each inner CV's training set.( I.e. you're saying you're doing CV correctly.) $\endgroup$ – Wayne May 31 at 14:16
1
$\begingroup$

You can use the defaults. Of course, it's possible to overfit to the test set by trying lots of different hyperparameter values and seeing what performance on the test set they lead to, but if all you're trying is two options, the default values and tuned values, and the default values themselves weren't set with the test set somehow, this kind of overfitting is not a substantial danger.

Note that when tuning hyperparameters worsens test-set performance but improves training-set performance, that's a hint that the tuning procedure is overfitting on the training set. Using regularization or switching to a simpler model may help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.