1
$\begingroup$

When performing k-fold cross-validation on a training set, what is the best way to handle a minimum cv error that occurs for multiple sets of different parameters.

Should all optimal parameters be used on the testing data set, and then the mean reported? Or are there some other options?

I feel that it is perhaps unfair to select the set of parameters that yield the lowest error on the testing data set as this will introduce information that is not available (i.e. I do not know beforehand which set will produce the minimum error on the testing data)

$\endgroup$

2 Answers 2

1
$\begingroup$

You have to select one optimal parameter set, that's the idea behind cross-validation. You use your test data set to estimate the generalized error of the model, and you can mix both strategies into what's known as nested cross validation. Now, what's optimal might depend on your needs.

In the case of perfect or technical draws between two or more parameter sets you can set a rule to select the optimal set. I can't really list all the possible ways of doing this but two examples include:

  • Select the one with the smallest resampled variance (i.e. the most consistent among all the candidates)

  • Select the simplest one (this can be defined by you or using other rules such as the degrees of freedom of each model, parametric vs. non-parametric model, runtime, etc)

You can also mix these with hypothesis testing, for example. The important part is only one parameter set should be tested.

$\endgroup$
1
$\begingroup$

There are different possible reasons why you observe (almost) the same performance for several parameter sets.

  • Your data set is small, so testing variance is large, and you randomly observe the same performance with your test set even though the actual performance of the tested models does vary.
    This is of course not desirable, it means that you have too few cases to afford data-driven hyperparameter optimization. In this case, you need to look for other ways (without using your data) to fix the hyperparameters. Or far more data. Or even a different model where hyperparameters are easily determined.
  • You found a parameter region where the model is rather insensitive to changes in the hyperparameters. I.e. the actual performance of the tested models is equal.
    In this case: congratulations, you can basically choose any parameter set in this region. I'd go for a central one.

In order to get an idea of what is going on, estimate confidence intervals for your observed performance and compare the observed differences as well as the precision that is acceptable for your application to the confidence interval.


Should all optimal parameters be used on the testing data set, and then the mean reported?

Rather than averaging, comparing the predictions for all cases and you set of "optimal" models can indicate whether you really found a hyperparameter region that yield good and stable models: in that case, the predictions for the same test case of all the models should be equal.

This would be a non-standard way of reporting model performance, but it would make sense.


I feel that it is perhaps unfair to select the set of parameters that yield the lowest error on the testing data set as this will introduce information that is not available (i.e. I do not know beforehand which set will produce the minimum error on the testing data)

Absolutely: the testing data should see the final model and treat it as a black box. No more fine tuning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.