I am writing a paper where I will analyze 6 designs. Responses for each design can be sampled from the main dataset. My question is that do I set 1 random seed for each sampling or 1 random seed for whole and sample them in the order of the design? A) set random seed(1) and then sample design I, set random seed(2) and then sample design II, ... B) set random seed(1) and then sample design I, design II ... design VI?
-
2$\begingroup$ Could you elaborate on what you mean by "appropriately"? What purpose(s) do you hope to achieve by setting the seeds? If it's only reproducibility, then in what sense is either algorithm not reproducible? $\endgroup$– whuber ♦Sep 18, 2018 at 21:42
-
$\begingroup$ @whuber What I meant "appropriately" means is there a standard way of using it in regards to reproducibility. $\endgroup$– Moses KimSep 18, 2018 at 21:44
-
1$\begingroup$ The algorithms you describe are a little unclear to me. Could you post some simple code / pseudocode to illustrate what you have in mind? $\endgroup$– gung - Reinstate MonicaSep 19, 2018 at 1:26
3 Answers
I recommend setting a separate seed each time you analyze a new design.
Why? For convenience and debugging. When you debug design IV and set a seed for this specific design, you can debug it easily. If you have set a "global" seed, then each time you want to do something to design IV that relies on a specific random number, you first have to run though designs I-III.
In addition, this allows you to add designs IIa, IIIa and IIIb at their places without disturbing the replicability of the results you already got for design IV. With a "global" seed, you would need to add designs IIa, IIIa and IIIb at the end of your code, at the cost of legibility.
Also: try your experiments with different seeds. If the conclusions change appreciably depending on the seed, something is likely broken, e.g., you may need more data to start with. Thus, if the answer to this question makes a substantive (as opposed to ease of programming related) difference, you are doing something wrong. See also If so many people use set.seed(123) doesn't that affect randomness of world's reporting?
You set the seed for the whole session and then sample designs I,II, and so forth. The use of a seed is essential if one wants to ensure the reproducibility of results.
For example, the R
random number generator function set.seed()
would allow you to restart the sequence by simply resetting the seed to the same value that was used previously. For example
set.seed(5)
X <- rnorm(10)
design1 <- sample(X)
design2 <- sample(X)
will yield the exact same sequences design1
and design2
every time you run this code, guaranteeing reproducibility of your results.
A random seed should be set before every run of a data-generating process.
I'm not familiar with some of the terminology in your question, so I'm going to make up a common scenario: simulating data to test your model on in R.
N <- 500
x1 <- rnorm(N)
x2 <- rnorm(N)
b1 <- runif(1, -1, 1)
b2 <- runif(1, -1, 1)
eps <- rnorm(N, 0.1)
y <- b1*x1 + b2*x2 + eps
In this case, you would set the seed before the first rnorm
.
Note that there is room for opinion here. If you are doing multiple simulations, it's easier to set the seed at the beginning and forget about it. But when you are doing interactive work, that makes your session hard to reproduce, because not every draw you make will be preserved in the code.