0
$\begingroup$

i'm new to cross validation approach and i want to know if the steps that i'm taking for my regression problem is correct:

  1. i am using nested cross validation for evaluating different algorithms(linear regression, random forest, ...) to find the winning algorithm.
  2. after which i will choose the winning algorithm and use Grid Search to find the best model in regard to parameter tuning. (grid.best_estimator_)
  3. i apply the wining model in step 2 to the whole Dataset.
$\endgroup$

1 Answer 1

0
$\begingroup$

Steps 1 and 2 have to be merged into one step. You can't evaluate different algorithms against each other without knowing what hyper-parameters to use for those algorithms.

I am assuming that in step 1 you are using default parameters, so maybe linear regression beats random forest with default parameters, but maybe random forest with a different value for mtry would have beat linear regression.

$\endgroup$
3
  • $\begingroup$ but nested cross validation is a combination of both itself and i am testing for different parameters but in nested cross validation with 10 fold i would have 10 different hyper-parameters for each algorithm and it is my understanding that i shouldn't use any of those, so i thought i should get the best one with another grid search after wards. $\endgroup$
    – john d
    Jul 13, 2018 at 15:52
  • 1
    $\begingroup$ Ahh okay. Then yes that is correct, I assumed that you weren't doing any tuning in step 1. Since you are, I believe that your process is correct, just make sure that you maintain the estimate of the generalization error that you found in step 1, not the one you get from step 2. $\endgroup$
    – astel
    Jul 13, 2018 at 15:55
  • $\begingroup$ You should check stability of the optimization in step 1, though. And give a glance at the hyperparameters for the final model to check they are not totally different from what you got in 1. $\endgroup$ Jul 15, 2018 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.