2
$\begingroup$

Suppose we want to tune the hyperparameters of an algorithm. We perform $k$-fold cross validation and we found the optimal hyperparameter values, lets say $p^*$. Nevertheless, we don't have a separate test. Can we report the average error from folds as an estimate of the generalization error?

Based on this Question we shouldn't.What I can't understand is the following.

Lets say that another analyst wants to estimate the generalization performance of $p^*$. He performs $k$-fold cv with the same data we used for hyperparameter tuning. He doesn't want to tune parameters. Just by chance he selected $p^*$ and wants to get an estimate of its performance. Further, lets assume that his $k$-folds are same to our $k$-folds. Obviously the average cv error would be the same. Why he can report this error as an estimate of the generalization error while we are not allowed?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Because you used the validation folds data to choose hyperparameters which minimise the apparent errors, and part of this process may have fitted noise in a way which biases the results downwards $\endgroup$
    – Henry
    Commented Nov 8, 2022 at 12:24
  • 1
    $\begingroup$ @Henry Can't the same being said for the analyst who just wants to estimate performance? $\endgroup$
    – Antonios Sarikas
    Commented Nov 8, 2022 at 12:41
  • $\begingroup$ That depends on how and why that analyst chose those hyperparameters. If they did it because of a different study using different data suggested this then they are fine as the new data is in effect a test set. If they did it because you suggested the model and hyperparameters using the same data as they are planning to use, then they are in effect you and you will get the same bias. $\endgroup$
    – Henry
    Commented Nov 8, 2022 at 12:57

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.