I am reading the book "An Introduction to Statistical Learning" available here.
Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use all of the predictors at each step). It does so for a regression problem. It then calculates out-of-sample MSE using a separate test set.
The code goes like that:
library(MASS) library(randomForest) set.seed(1) train = sample(1:nrow(Boston), nrow(Boston)/2) # Bagging = Bootstrap aggregation bag.boston = randomForest(medv~., data=Boston, subset=train, mtry=13, importance=T) yhat <- predict(bag.boston, newdata=Boston[-train,]) y <-Boston[-train,]$medv mse = mean((y-yhat)^2) print(paste0("MSE: ", mse))
To be truth, at each step the training set is bootstrapped, and the decision tree is trainined on the bootstrapped dataset. Some samples are not drawn during bootstrapping, and are used after training to compute MSE. At least this is how I understood it.
Why the book uses a separate validation set, instead of OOB samples, to compute out-of-sample MSE?
Said otherwise, why the following is wrong?
I have tried it, and the results are as follows. The code:
mse = mean((y-yhat)^2) print(paste0("MSE: ", mse))
19.88, while the code
returns a much lower
11.72. So it seems that the latter is overfitted, but I don't understand why.