Can the $R^2$ measure be used to measure the performance of Random Forest model? My explanatory and dependent variables are linearly dependent.

  • $\begingroup$ This has been written about before on this site. Check out, for instance, stats.stackexchange.com/q/13869/40036 $\endgroup$ – josliber Dec 11 '20 at 19:39
  • $\begingroup$ Why are you using a random forest of the relationship between your variables is known to be linear? $\endgroup$ – Dave Dec 11 '20 at 19:58
  • $\begingroup$ @Dave At some points it might not yield a linear model. $\endgroup$ – Agi Dec 11 '20 at 20:29
  • $\begingroup$ @josliberI did actually but it doesn't give a straight answer of if it is theoretically wrong to do it. $\endgroup$ – Agi Dec 11 '20 at 20:29

There is no advantage over using MSE or RMSE. Any model that has better $R^2$ than another model also will have better MSE and RMSE (assuming the same data). In that sense, all three are equivalent loss functions and measures of performance.

A common reason for wanting to use $R^2$ over MSE or RMSE is the desire to say that $R^2=0.94$ means that $94\%$ of the variation in the data is explained by the model, and $94\%$ is an $\text{A}$ grade in school. This interpretation of $R^2$ fails for nonlinear models like random forest, as the residuals and predictions are not orthogonal. (That there is a linear relationship between your variables is not relevant to this point.)

So you can use $R^2$ when MSE or RMSE would be viable loss functions, but I don’t see a reason to do so.

  • $\begingroup$ Are you trying to compare the RF performance to the linear regression $R^2& value? $\endgroup$ – Dave Dec 11 '20 at 19:59
  • $\begingroup$ Yes. I want to compare the performance of different models with one another. The concern that I have is that RF might build a non-linear model for some training sets (I'm using rolling time window), that's why I wonder if it makes sense to use rf for non-linear mdoels. $\endgroup$ – Agi Dec 11 '20 at 20:25
  • $\begingroup$ I do not follow your concerns. Perhaps post them as a new question. It looks like it will be interesting to discuss. $\endgroup$ – Dave Dec 11 '20 at 20:28
  • $\begingroup$ Could you provide the formula for MSE to be used for non-linear models? Is it $MSE=\frac{1}{n}\Sigma_{i=1}^n(y_i-\hat{y_i})^2$ for all sort of models? $\endgroup$ – Agi Dec 13 '20 at 18:15
  • 1
    $\begingroup$ @Agi The formula for MSE does not depend on the model, so what you wrote applies in general. $\endgroup$ – Dave Dec 13 '20 at 18:29

If you take the definition of $R^2$ coming as a log-likelihood difference ratio (see this answer) then it is a perfectly valid choice. Under this framework, you can interpret $R^2$ as the fraction of explained deviance.

  • 1
    $\begingroup$ Ben’s answer seems to be specific to GLMs, though I should keep it in mind if I want some kind of scaled performance metric for a logistic regression. $\endgroup$ – Dave Dec 11 '20 at 20:10
  • $\begingroup$ @Dave it's actually valid to any likelihood of choice, since it's simply a comparison of likelihoods. GLMs do, however, maximize that quantity directly, if the respective distribution conforms to the GLM framework $\endgroup$ – Firebug Dec 11 '20 at 20:52
  • $\begingroup$ @Firebug Do non-linear models fit into GLM category? Cause the way it computes the r-squared leads down to the same traditional r-squared definition of the linear models. $\endgroup$ – Agi Dec 11 '20 at 21:06
  • $\begingroup$ @Agi Your random forest is not a GLM, no. $\endgroup$ – Dave Dec 11 '20 at 23:11
  • 1
    $\begingroup$ @Agi you’re mixing up two questions. One is if random forest is a generalized linear model (GLM), and the answer is that it is not. The other is about how proportion of deviance explained works in general, and I will let Firebug address that. $\endgroup$ – Dave Dec 12 '20 at 3:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.