0
$\begingroup$

i'm new to cross validation approach and i want to know if the steps that i'm taking for my regression problem is correct:

  1. i am using nested cross validation for evaluating different algorithms(linear regression, random forest, ...) to find the winning algorithm.
  2. after which i will choose the winning algorithm and use Grid Search to find the best model in regard to parameter tuning. (grid.best_estimator_)
  3. i apply the wining model in step 2 to the whole Dataset.
$\endgroup$
0
$\begingroup$

Steps 1 and 2 have to be merged into one step. You can't evaluate different algorithms against each other without knowing what hyper-parameters to use for those algorithms.

I am assuming that in step 1 you are using default parameters, so maybe linear regression beats random forest with default parameters, but maybe random forest with a different value for mtry would have beat linear regression.

$\endgroup$
  • $\begingroup$ but nested cross validation is a combination of both itself and i am testing for different parameters but in nested cross validation with 10 fold i would have 10 different hyper-parameters for each algorithm and it is my understanding that i shouldn't use any of those, so i thought i should get the best one with another grid search after wards. $\endgroup$ – john d Jul 13 '18 at 15:52
  • 1
    $\begingroup$ Ahh okay. Then yes that is correct, I assumed that you weren't doing any tuning in step 1. Since you are, I believe that your process is correct, just make sure that you maintain the estimate of the generalization error that you found in step 1, not the one you get from step 2. $\endgroup$ – astel Jul 13 '18 at 15:55
  • $\begingroup$ You should check stability of the optimization in step 1, though. And give a glance at the hyperparameters for the final model to check they are not totally different from what you got in 1. $\endgroup$ – cbeleites supports Monica Jul 15 '18 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.