I'm using randomForest to fit a model with continuous response variable. I was reading the An Introduction to Statistical Learning: with Applications in R (Springer Texts in Statistics), in Chapter 8, section 8.3.3 Bagging and Random Forests, it uses the following example: randomforest_code

I followed the exact same command and got the following result: enter image description here

The MSE from summary is 14.5. The textbook then used the following formula to calculate MSE in test set: enter image description here

Instead of the test set, I used this formula to calculate the MSE for the training set (the set I used to obtain the model), and here's my code: enter image description here

However, as you can see, the outcome is way different from the result from summary statistics. I'm not sure why this happens, and which one is the correct MSE, and if I want to use MSE to compare to compare with other models, which MSE should I use?

Thank you in advance.

  • $\begingroup$ The output in the summary when you print the object uses only out of bag observations, so that you aren't evaluating the model on observations used to fit the model. $\endgroup$ – joran Nov 30 '15 at 2:41
  • 1
    $\begingroup$ Both are "correct" but the MSE when calculated on the training set is a biased estimate of the true MSE $\endgroup$ – joran Nov 30 '15 at 2:45
  • $\begingroup$ I used the same formula (sum of squared residual/n) to calculate MSE for other models I tried, such as MLR, regression tree with bagging, although it is biased, can I use it to compare which model gives better fit? or should I use the one the summary statistics gives? $\endgroup$ – ttothef Nov 30 '15 at 2:50
  • 1
    $\begingroup$ No, that is, in general, very bad. The accuracy of models should only ever be assessed on new data, or procedures that approximate new data (cross validation, bootstrapping, out of bag, etc) $\endgroup$ – joran Nov 30 '15 at 2:52
  • $\begingroup$ This really isn't a programming question and therefor doesn't really belong on Stack Overflow. If you have questions about the proper interpretation of statistical models, you should ask at Cross Validated where such questions are on-topic. $\endgroup$ – MrFlick Nov 30 '15 at 3:12

The textbook is comparing the random forest predicted values against the real values of the test data. This makes sense as a way to measure how well the model predicts: compare the prediction results to data that the model hasn't seen.

You're comparing the random forest predictions to a specific column of the training data. I don't understand why this would be helpful: you're comparing predictions for observations in the training set to some value of observations in the test set. That's like taking the average height of yourself and your neighbor. Sure, you can compute it, but how is that helpful?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.