3
$\begingroup$

Consider the following scenario where you use the same data X (the same number of predictors p, same number of observations n) to predict a continuous outcome y, in 2 different regression models (e.g. Linear Regression and Random Forests). For both models you calculate RMSE and R-squared (assume simple correlation between y and y-hat, squared).

You get:

RMSE1 < RMSE2

R-squared1 < R-squared2

Could someone explain if this scenario is possible and how it occurs? A simple simulation in R could also do.

$\endgroup$
1
$\begingroup$

This may arise if the regression model is nonlinear. Consider the following true data

0.6
0.55
0.3
0.24
0.22

With model 1 predicting

0.82
0.72
0.55
0.29
0.1

and model 2

0.7
0.58
0.51
0.2
0.09

$R^2$ for Model 1 is 0.8448 while for Model 2 is 0.7866.

However, Model 1 has higher RMS error, at 0.1770 but Model 2 only has RMSE of 0.1212

This happens because $R^2$ is only really valid for linear regression models (and data).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.