4
$\begingroup$

I'm using randomForest to fit a model with continuous response variable. I was reading the An Introduction to Statistical Learning: with Applications in R (Springer Texts in Statistics), in Chapter 8, section 8.3.3 Bagging and Random Forests, it uses the following example: randomforest_code

I followed the exact same command and got the following result: enter image description here

The MSE from summary is 14.5. The textbook then used the following formula to calculate MSE in test set: enter image description here

Instead of the test set, I used this formula to calculate the MSE for the training set (the set I used to obtain the model), and here's my code: enter image description here

However, as you can see, the outcome is way different from the result from summary statistics. I'm not sure why this happens, and which one is the correct MSE, and if I want to use MSE to compare to compare with other models, which MSE should I use?

Thank you in advance.

Improve this question
$\endgroup$
7
  • $\begingroup$ The output in the summary when you print the object uses only out of bag observations, so that you aren't evaluating the model on observations used to fit the model. $\endgroup$
    – joran
    Commented Nov 30, 2015 at 2:41
  • 1
    $\begingroup$ Both are "correct" but the MSE when calculated on the training set is a biased estimate of the true MSE $\endgroup$
    – joran
    Commented Nov 30, 2015 at 2:45
  • $\begingroup$ I used the same formula (sum of squared residual/n) to calculate MSE for other models I tried, such as MLR, regression tree with bagging, although it is biased, can I use it to compare which model gives better fit? or should I use the one the summary statistics gives? $\endgroup$
    – ttothef
    Commented Nov 30, 2015 at 2:50
  • 1
    $\begingroup$ No, that is, in general, very bad. The accuracy of models should only ever be assessed on new data, or procedures that approximate new data (cross validation, bootstrapping, out of bag, etc) $\endgroup$
    – joran
    Commented Nov 30, 2015 at 2:52
  • $\begingroup$ This really isn't a programming question and therefor doesn't really belong on Stack Overflow. If you have questions about the proper interpretation of statistical models, you should ask at Cross Validated where such questions are on-topic. $\endgroup$
    – MrFlick
    Commented Nov 30, 2015 at 3:12

2 Answers 2

Reset to default
1
$\begingroup$

The textbook is comparing the random forest predicted values against the real values of the test data. This makes sense as a way to measure how well the model predicts: compare the prediction results to data that the model hasn't seen.

You're comparing the random forest predictions to a specific column of the training data. I don't understand why this would be helpful: you're comparing predictions for observations in the training set to some value of observations in the test set. That's like taking the average height of yourself and your neighbor. Sure, you can compute it, but how is that helpful?

Improve this answer
$\endgroup$
0
$\begingroup$

The reason is that you are calculating your MSE using the predictions for data.train that are obtained considering all the resulting model of combining each tree. However the Random Forest is calculating the MSE using the predictions obtained from evaluate the same data.train in every tree but only considering the data is not taken from bootstrapping to construct the tree, wether the data that it is in the OOB (OUT-OF-BAG). Then it averages the predictions for all the OUT-OF-BAG predictions for each sample of data.train that comes as input.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.