1
$\begingroup$

I ran a regression with tidymodels following this following along with the random forest example here but using different data.

When I ran it with four variables or so, I got an R Squared of 0.94 but a high (for what I’m trying to predict) RMSE of 20000. I added more variables and got an R Squared of 0.97 and RMSE of 40000. Why would the RMSE increase if the R Squared supposedly indicated the model was better? I believe RMSE means how far off my prediction is from the actual test data. I’m trying to bring my RMSE down which is why I added more variables.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ Is your RMSE measured in-sample, or on a holdout sample, or on an out-of-bag sample? $\endgroup$
    – Stephan Kolassa
    Commented Dec 30, 2021 at 23:56
  • 1
    $\begingroup$ You are correct to think that $R^2$ increasing corresponds to $MSE$ decreasing…if you are measuring those values on the same data. It is absolutely possible, however, that in-sample $R^2$ increases while out-of-sample $MSE$ also increases, (Indeed, we use the out-of-sample metric plot detect such behavior.) Such a situation corresponds to in-sample MSE decreasing but out-of-sample MSE increasing. $\endgroup$
    – Dave
    Commented Dec 31, 2021 at 1:13
  • 4
    $\begingroup$ Because this sounds mathematically impossible, please supply a reproducible example. $\endgroup$
    – whuber ♦
    Commented Dec 31, 2021 at 2:02
  • $\begingroup$ I think it’s in-sample @Stephan Kolassa $\endgroup$
    – Learning_and_xbox
    Commented Dec 31, 2021 at 3:49

1 Answer 1

Reset to default
1
$\begingroup$

One visual interpretation of adding more variables to your model, which implies adjusting for them and thus slicing your data, is that you may end up with only the points that are closer to the curve you're fitting. This explains why your $R^2$ is increasing when you add more variables. Indeed, your model fits better the data [that is left], however, this may not always be the case. You can adjust for new variables and get a worse $R^2$, by having data points that make it hard to fit with the curve you're trying to use. Besides, this does not prevent overfitting. If your model is overfitting, you can still get a high $R^2$ and very large RMSE when assessing your model on test data (data that was not used to build your model).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Caeteris paribus, $R^2$ increases as the RMSE decreases: this is what requires an explanation. $\endgroup$
    – whuber ♦
    Commented Dec 31, 2021 at 2:01
  • 1
    $\begingroup$ Thank you all for your replies. I am examining the RMSE on test data after training. It does increase along with the R Squared. I will look into overfitting and providing a reproducible example tomorrow. $\endgroup$
    – Learning_and_xbox
    Commented Dec 31, 2021 at 3:43
  • 1
    $\begingroup$ $R^2=1-\dfrac{RMSE^2}{\sum\big(y_i-\bar y\big)^2}$ so bigger $RMSE \iff$ smaller $R^2$ and smaller $RMSE\iff$ bigger $R^2$, since that denominator term is the same for every model of the same data. $\endgroup$
    – Dave
    Commented Dec 31, 2021 at 3:43
  • $\begingroup$ Should be $n\times RMSE^2$ in the numerator of my earlier comment $\endgroup$
    – Dave
    Commented Dec 31, 2021 at 3:56
  • $\begingroup$ @Learning_and_xbox, are RMSE and $R^2$ calculated on the exact same data? You said RMSE comes from test data. Where does $R^2$ come from? $\endgroup$
    – Richard Hardy
    Commented Dec 31, 2021 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.