3
$\begingroup$

I try to do a ridge and lasso regression for out of sample predictions. The optimal lambda is chosed via cross validation. I run my results for different seeds in R. And depending on the seed i get a different result.

Now i ask myself do efficiency of this shrinkaged methods realy depend on the seed? I mean do i have to find the right seed to produce an modell that gives me the smallest mse?

$\endgroup$
4
$\begingroup$

Each time you run a cross validation you get an estimate of the true population test error. Different random seeds give you different estimates of this population quantity, but each is an estimate of the same underlying truth.

If you re-seed the random number generator in a quest to drive down the estimated test error you are going to bias your estimate of the test error downwards (this is sometimes called seed-hacking, and is a bad practice). If you take your final test error estimate to be the minimal you have seen over many runs of the random number generator, then you're definitely going to end up with a very optimistic estimate of the test error, and should not be surprised when your model performs much worse in production.

Instead, pick a seed and stick with it. My personal seed is 154.

$\endgroup$
  • $\begingroup$ Personally, I roll several 10-sided die each time I need to make a reproducible seed. $\endgroup$ – Sycorax Jul 9 '18 at 20:02
  • $\begingroup$ The real question is if you sum the results or concatenate them... $\endgroup$ – Matthew Drury Jul 9 '18 at 20:21
  • 1
    $\begingroup$ Lol. Concatenate. $\endgroup$ – Sycorax Jul 9 '18 at 20:23
  • $\begingroup$ @MatthewDrury maybe i can solve the problem with the seed if is use reapeted cross validation? $\endgroup$ – Dima Ku Jul 10 '18 at 9:13
  • $\begingroup$ I'm unsure what problem you are solving with that? Are you looking for an estimate of the variance of your test error estimate? $\endgroup$ – Matthew Drury Jul 10 '18 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.