1
$\begingroup$

I have a machine learning algorithm with some hyperparameters. First, I split the data to 70% (A-set) and 30% (B-set).

Then, I used 5-fold cross-validation on the A-set to find the best hyperparameters.

Finally, I used 10-fold cross-validation on all data for reporting the performance of the algorithm.

Was my approach correct? If yes, is there any reference for it?

Is my approach biased?

Thanks in advance.

$\endgroup$
1

1 Answer 1

0
$\begingroup$

This approach is not correct - hyperparameters are tuned on the data that is later used for evaluation. A better approach would be to only test on the set B using 10 fold.

$\endgroup$
1
  • 1
    $\begingroup$ Or, instead of a single split into A and B sets, go for nested cross validation, i.e. wrap the 10-fold CV around the optimized model training with the 5-fold-CV inside. $\endgroup$ Commented Jul 22, 2019 at 10:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.