0
$\begingroup$

I am having trouble to understand where and how many times to use set.seed().

I am performing a leave-one out crossvalidation on a set with 120 classified variables to predict another 300 of unknown class. Should I set the seed at the beginning of the code or at the beginning of each iteration of the leave-one-out cross validation in order to guarantee a run that can be repeated?

Thanks for any help and explanations.

$\endgroup$
3
$\begingroup$

If you set it once, the run will be replicable. Setting it multiple times will, at best, just slow things down. At worst, it would no longer be random. (It's hard to tell without your actual code).

$\endgroup$
3
$\begingroup$

Are you really doing leave-one-out cross validation?

Leave-one-out cross validation is usually complete in the sense that one iteration (consisting of $n$ surrogate models tested with one left out case each) covers all possible surrogate model of $n - 1$ cases plus the one test case combinations. Doing another iteration of this will result in exactly the same predicitons unless your classifier is non-deterministic.

In the case of a completely deterministic classifier the random number generator is not needed at all. If your classifier uses the random number generator, you can set the seed at the beginning of the cross validation.
If I'd want to be able to exactly reproduce arbitrary surrogate models, I'd log the state of the random number generator just before the call to the classifier rather than setting it.


Note that while the result of leave-one-out cross validation of a deterministic classifier will be exactly the same for each iteration, it is still subject to variance due to the finite number of tested cases and instability of the surrogate models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.