How does Random Number Generation work?

Question

I'm using the set.seed() function in R to achieve reproducability of my results. I compare different regression methods (e.g. RandomForest, SVM, GAM) by their MSE derived from a cross-validation procedure. To my surprise, I realized that results differ whether I place 'set.seed(123)' at the beginning of my code (and then running the whole script) or whether I place 'set.seed(123)' just before calling each method in the script.

To illustrate pls follow my example below (although the answer by 'Sean Easter' and the example given by 'Cliff AB' below should explain as well):

data(iris)
iris
myf<- Sepal.Length ~ 
 Sepal.Width+
 Petal.Length+
 Petal.Width+
 Species

# required packages
library(sperrorest)
library(randomForest)
library(rpart)

##### Regression Tree
set.seed(123)
ctrl <- rpart.control(cp = 0.001)
fit_rpart <- rpart(myf, data = iris, control = ctrl)

#5-repeated 10-fold CV
mypred.rpart <- function(object, newdata) predict(object, newdata)
eval_ns_rpart <- sperrorest(data = iris, formula = myf, model.fun= 
                            rpart, model.args = list(control = ctrl),
                            pred.fun = mypred.rpart, smp.fun = 
                            partition.cv, smp.args = 
                            list(repetition=1:5, nfold=10))
summary(eval_ns_rpart$error)

##### Random Forest
#set.seed(123) # REMOVE HASH IN 2ND RUN!!!! 
fit_rf <- randomForest(myf, data = iris, ntree=1000)

#5-repeated 10-fold CV
mypred.rf <- function(object, newdata) predict(object, newdata)
eval_ns_rf <- sperrorest(data = iris, formula = myf,
                         model.fun = randomForest,
                         pred.fun = mypred.rf,
                         smp.fun = partition.cv, smp.args= list(repetition=1:5, nfold=10))
summary(eval_ns_rf$error)

#### SUMMARIES Mean Squared Errors(MSE)
tr_MSE_rpart<-(summary(eval_ns_rpart$error)[3,1]) # MSE training error
# 0.08548725

t_MSE_rpart<-(summary(eval_ns_rpart$error)[10,1]) # MSE test error
# 0.1445583

tr_MSE_RF<-(summary(eval_ns_rf$error)[3,1]) # MSE training error
# 0.07241344 # 2nd run: 0.07266605

t_MSE_RF<-(summary(eval_ns_rf$error)[10,1]) # MSE test error
# 0.1403778  # 2nd run: 0.1358957

Answer 1

Plainly, pseudorandom number generators are cyclical. Setting the seed chooses where in their cycle they will begin. From wiki:

A PRNG can be started from an arbitrary initial state using a seed state. It will always produce the same sequence when initialized with that state.

So, when you call set.seed in R, you're resetting the generator to the seed state, moving it from whatever particular point it held in the cycle.

Answer 2

You seem to be confused about what set.seed does. To help illustrate, try the following code.

set.seed(123)
rnorm(5)
rnorm(5)

set.seed(123)
rnorm(5)
rnorm(5)

To extend this example to something closer to what you are doing, also run:

set.seed(123)
rnorm(5)
set.seed(123)
rnorm(5)

and note that the 5 random numbers in each case are the same (and are the same as the first 5 numbers generated after each call to set.seed(123) in the first example).