How to interpret R-squared versus nRMSE for a random forest model?

Question

I have trained six random forest regression models (to predict topsoil, subsoil and total soil organic carbon stocks for two study ares) using out-of-bag validation, and I have gathered the R² and normalised RMSE (normalised using the mean of the measured values) of the validation, as suggested by my thesis supervisors. The results can be seen in the table at the bottom of the post.

I am a bit lost on how these two accuracy metrics relate to each other and how they should be interpreted. From what I've read online, the R² quantifies how much of the variation is explained by the model while the RMSE quantifies how much of the variation is left unexplained, so if I understand correctly, R² and RMSE should tell two sides of the same story. Then how is it possible that the R² and nRMSE of my models are so inconsistent? (i.e. the "main" models have consistently higher R² values than the "valley" models, but the nRMSE values of both are similar and even a bit lower for the "valley" models.) Am I not interpreting them right? Does it have something to do with the normalisation of the RMSE?

Answer 1

$R^2$ only has the "proportion of variance explained" interpretation under particular circumstances, and a random forest is not such a circumstance. Here, I explain the math of why this is the case. In the language of that answer, the $Other$ term is not zero in a random forest.

Consequently, any intuition about $R^2$ and the "proportion of variance explained" is gone.

However, the random forest $R^2$ and the $nRMSE$ are transformations of the mean squared error that attempt to give context to the mean squared error. For $R^2$, there are various interpretations, but the idea is to transform the $MSE$ like so:

$$ R^2= 1 -\left( \dfrac{MSE}{C} \right) =1-\left(\dfrac{ \dfrac{1}{N}\overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ C }\right) $$

Depending on your philosophy, you might pick different values of $C$ (this is where sklearn and I disagree, if you read the link). I consider my interpretation to be a comparison to a baseline model, where $1$ indicates a perfect fit, $0$, indicates predictions no better (in terms of mean squared error) than if you predicted the (in-sample) mean value of $y$ every time, values less than $0$ indicate a poor fit that is outperformed by naïvely predicting the (in-sample) mean of $y$ every time, and values in $(0,1)$ indicate various degrees of outperforming that naïve prediction of the same value every time, with higher being better. If you take a different value of $C$, then you might be able to wrestle out another interpretation (and I would welcome a post about that interpretation as an answer to my linked question about sklearn).

For $nRMSE$, the normalization happens by taking the square root of the MSE (the RMSE) and then dividing by the mean of the $y$ values. This gives context based on the size of a typical value. The thinking is that larger errors is more acceptable for larger average values. Think about it this way: if you are off by $\$100$ on a restaurant bill, that is a large error, perhaps more than the entire bill, but if you are off by $\$100$ on the price of a house, such an error is less meaningful, since houses cost so much more than dinner out.

$$ nRMSE = \dfrac{RMSE}{\bar y} = \dfrac{\sqrt{MSE}}{\bar y} = \dfrac{ \sqrt{\dfrac{1}{N} \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 } }{ \bar y } $$

Notice that this $nRMSE$ is problematic if the mean value of $y$ is zero, and if $\bar y<0$, then I really have no idea what to make of it. For quantities that are always positive, however, such as distances or stock prices (but not financial returns), this $nRMSE$ could be useful. A related measure of performance is the mean absolute percentage error ($MAPE$), which has a Wikipedia page that explains what it is (definition is not so surprising) and goes through the considerable downsides it has, despite what appears to be an appealing interpretation. I have my doubts that $nRMSE$ evades all of these criticisms of $MAPE$.