2
$\begingroup$

I am using Random Forests in Matlab for regression. After educating my model on train data, I want to get MSE on test data not used in training. I do that two ways:

  1. call predict and directly calculate MSE using predicted and actual values
  2. call error and use built in TreeBagger functionality to do the same task.

In first case I get 10 times bigger result. Why? The only explanation I have, is that built in function somehow discounts outliers in prediction, but I am not sure how exactly it is done.

Can somebody, please, explain all this to me.

$\endgroup$
2
$\begingroup$

If you use predict on a train set, the result will be highly overfitted and meaningless -- this actually applies to all ML algorithms, and this is why test sets, cross-validations and similar stuff is used.

However, one of the big advantages of bagging is that it can produce an approximation of error on train set as it would be a test -- this is called out-of-bag (OOB) and those predictions (available in OOBPred) are used by Matlab in error to produce "true" error.

$\endgroup$
  • $\begingroup$ I am using predict on test set. It's written in my question. From Matlab help: "err = error(B,X,Y) computes mean squared error (MSE) for regression trees for each tree, for predictors X given true response Y." What you are talking about is (again Matlab reference quote):"err = oobError(B) computes mean squared error (for regression trees) for out-of-bag observations in the training data". $\endgroup$ – GrayR Oct 2 '12 at 19:35
  • $\begingroup$ Okay, it's really shamefull but I must admit that difference between Matlab built in function and my direct calculations appeared due to two factors: 1) typo in my MSE formula, 2) setting mode 'individual' in error command and getting array of errors per tree in forest, instead of total error. I will accept mbq's answer because actually his comment lead me to check TreeBagger's options one more time and find my mistakes. $\endgroup$ – GrayR Oct 2 '12 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.