1
$\begingroup$

I'm using the set.seed() function in R to achieve reproducability of my results. I compare different regression methods (e.g. RandomForest, SVM, GAM) by their MSE derived from a cross-validation procedure. To my surprise, I realized that results differ whether I place 'set.seed(123)' at the beginning of my code (and then running the whole script) or whether I place 'set.seed(123)' just before calling each method in the script.

To illustrate pls follow my example below (although the answer by 'Sean Easter' and the example given by 'Cliff AB' below should explain as well):

data(iris)
iris
myf<- Sepal.Length ~ 
 Sepal.Width+
 Petal.Length+
 Petal.Width+
 Species

# required packages
library(sperrorest)
library(randomForest)
library(rpart)

##### Regression Tree
set.seed(123)
ctrl <- rpart.control(cp = 0.001)
fit_rpart <- rpart(myf, data = iris, control = ctrl)

#5-repeated 10-fold CV
mypred.rpart <- function(object, newdata) predict(object, newdata)
eval_ns_rpart <- sperrorest(data = iris, formula = myf, model.fun= 
                            rpart, model.args = list(control = ctrl),
                            pred.fun = mypred.rpart, smp.fun = 
                            partition.cv, smp.args = 
                            list(repetition=1:5, nfold=10))
summary(eval_ns_rpart$error)

##### Random Forest
#set.seed(123) # REMOVE HASH IN 2ND RUN!!!! 
fit_rf <- randomForest(myf, data = iris, ntree=1000)

#5-repeated 10-fold CV
mypred.rf <- function(object, newdata) predict(object, newdata)
eval_ns_rf <- sperrorest(data = iris, formula = myf,
                         model.fun = randomForest,
                         pred.fun = mypred.rf,
                         smp.fun = partition.cv, smp.args= list(repetition=1:5, nfold=10))
summary(eval_ns_rf$error)

#### SUMMARIES Mean Squared Errors(MSE)
tr_MSE_rpart<-(summary(eval_ns_rpart$error)[3,1]) # MSE training error
# 0.08548725

t_MSE_rpart<-(summary(eval_ns_rpart$error)[10,1]) # MSE test error
# 0.1445583

tr_MSE_RF<-(summary(eval_ns_rf$error)[3,1]) # MSE training error
# 0.07241344 # 2nd run: 0.07266605

t_MSE_RF<-(summary(eval_ns_rf$error)[10,1]) # MSE test error
# 0.1403778  # 2nd run: 0.1358957
$\endgroup$
  • 1
    $\begingroup$ Can you provide a reproducible example (simplified code & output)? Also, be aware that asking about how to use software is off topic here. At present, it isn't clear if this is a conceptual issue or an R issue; we may migrate this to Stack Overflow once the issue clarifies. $\endgroup$ – gung - Reinstate Monica Jan 20 '16 at 17:38
  • $\begingroup$ Imagine that a random number generator simply takes consecutive numbers from a very long list of jumbled-up numbers (it's not what happens, but it's a very good model for the process). When you first set the seed, you tell it where in the list to start from. But the next time you call the random number generator, it keeps going from where it got up to (it uses a number to keep track of where it was in the list). If you re-set the seed later, it forgets where it just was and starts again from where you tell it. If you keep that in mind while you look at your code, you'll see why it matters $\endgroup$ – Glen_b -Reinstate Monica Jan 20 '16 at 22:26
  • $\begingroup$ I think this could be rephrased to be about how random number generators work more generally (rather than R's random number generation specifically), which is arguably on topic for this site; .e.g if the R aspect was reduced to an example it might be able to be reopened. $\endgroup$ – Glen_b -Reinstate Monica Jan 20 '16 at 22:31
3
$\begingroup$

Plainly, pseudorandom number generators are cyclical. Setting the seed chooses where in their cycle they will begin. From wiki:

A PRNG can be started from an arbitrary initial state using a seed state. It will always produce the same sequence when initialized with that state.

So, when you call set.seed in R, you're resetting the generator to the seed state, moving it from whatever particular point it held in the cycle.

$\endgroup$
  • $\begingroup$ I wasn't aware about that! $\endgroup$ – wetterfrosch Jan 21 '16 at 12:22
2
$\begingroup$

You seem to be confused about what set.seed does. To help illustrate, try the following code.

set.seed(123)
rnorm(5)
rnorm(5)

set.seed(123)
rnorm(5)
rnorm(5)

To extend this example to something closer to what you are doing, also run:

set.seed(123)
rnorm(5)
set.seed(123)
rnorm(5)

and note that the 5 random numbers in each case are the same (and are the same as the first 5 numbers generated after each call to set.seed(123) in the first example).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.