# Will a model with a low R² value have higher RMS error rates?

I'm testing different models and my general expectation is that models that have a high coefficient of determination should roughly also have a lower error rate (RMSE in that case) than those with a low R² (please correct me if this premise is wrong).

I've defined three models, linear regression, decision trees and random forests:

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.tree import DecisionTreeRegressor

[some code missing here]

def get_rmse(testdata, reg):
errorsum = 0
items = 0
for index, row in testdata.iterrows():
y = row["Target"]
X = np.array(row)[:-1].reshape(1, -1)
y_hat = reg.predict(X)
errorsum += (y-y_hat.item())**2
items += 1
rmse = np.sqrt(errorsum/items)
return rmse

reg = LinearRegression().fit(X_train, y_train)
score = reg.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, reg)}")
print("-------")

dt = DecisionTreeRegressor()
dt.fit(X_train, y_train)
score = dt.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, dt)}")
print("-------")

rf = RandomForestRegressor(n_jobs=-1)
rf.fit(X_train, y_train)
score = rf.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, rf)}")


The results for my dataset are:

R²: 0.8322231990679154
RMSE: 1.6443917859748052
-------
R²: 0.967768714719696
RMSE: 3.0779040219133758
-------
R²: 0.9772274437532209
RMSE: 2.020003916126851


So it looks like the linear regression model's RMSE is very low but it also has a low R². Why? Shouldn't we expect that a model that makes less mistakes also explains the data better?

sklearn defines score to be the $$R^2$$:

$$R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}$$

Where $$\text{RSS} = n\text{MSE}$$ and $$\text{TSS} = (n-1)\operatorname{Var}(y)$$, the sample variance of $$y$$.

The RMSE on the other hand is:

$$\text{RMSE} = \sqrt{\text{MSE}}$$

So we can relate one to the other:

$$R^2 = 1 - \frac{\text{RMSE}^2}{\text{TSS}}$$

And

$$\text{RMSE} = \sqrt{(1-R^2)\text{TSS}}$$

So, if $$\text{RMSE}$$ increases, $$R^2$$ decreases and vice versa.

• That's also my expectation but aren't those value comparable over different models? If you compare linear regression and decision trees in my question, the linear regression model has a R² of 0.8 and a RMSE of 1.64. The decision tree has a R² of 0.97 so my expectation would be that the RMSE a decision tree model makes must definitely be lower than the RMSE of the linear regression, so lower than 1.64. – Christian Vorhemus Jan 13 at 18:24