19
$\begingroup$

It is standard advice to set a random seed so that results can be reproduced. However, since the seed is advanced as pseudo-random numbers are drawn, the results could change if any piece of code draws an additional number.

At first glance, version control looks to be a solution to this, as it would at least allow you to go back and reproduce the version extant when you wrote down the results in your notes or paper. However, since it only takes one draw to mess things up, if you update R the results could change as well.

I realize that this is probably only problematic in rare cases, but I'm curious if there are any best practices here. This is something I've been struggling with in my own work.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
15
$\begingroup$

It depends how you will run the code or if there is any code that is somewhat stochastic in that it draws random numbers in a random way. (An example of this is the permutation tests in our vegan package where we only continue permuting until we have amassed enough data to know whether a result is different from the stated Type I error tacking into account a Type II error rate.) Although even that shouldn't affect the draws...

If the final script will only ever be run as a batch job or in its entirety and there are no stochastic draws from the pseudo-random number generator then it is safe to set a seed at the top of the script and run it in its entirety.

If you want to step through code, perhaps rerunning blocks then you need a set.seed() call before each function call that will draw from the pseudo-random number generator.

For my scientific papers I routinely go super defensive and set seeds prior to each code chunk; this allows for updates to the script at a later date that might need to be inserted into the existing script at any point - say to respond to reviewers' or co-authors' comments.

Your results will hopefully not be contingent on a particular set of pseduo-random values, so the issue is being able to reproduce the exact values stated in a report or paper. Even though you might be super defensive and set a seed on each code chunk, you still may need to recreate the exact installation --- R version and package versions so recording those details is essential. To be extra safe you'll need to keep previous R versions and packages around for specific projects/papers. Indeed, many people do this.

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1. Re the last paragraph: you don't have to save all that junk and you don't have to recreate an entire installation. If you're specific about which RNG you use, then rather than accepting defaults, all that needs saving are (1) the source code for that RNG (which is usually short) and (2) the state of the RNG at each crucial juncture. For most R work this state can found in .Random.seed. My biggest concern about R is that some routines might circumvent this--and perhaps could ignore set.seed altogether in some cases. $\endgroup$
    – whuber ♦
    May 14, 2013 at 2:50
  • 2
    $\begingroup$ @whuber I was thinking more generally there - if the concern is reproducing the exact set of results you will more than likely need the version of R and versions of any packages used. To whit; R 3.0.0 changed the precision with which it reported values - not major but that was enough to throw off all many package check tests that were assuming too much precision. Also, packages are updated regularly and things change. $\endgroup$
    – Gavin Simpson
    May 14, 2013 at 3:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.