Will a model with a low R² value have higher RMS error rates?

Question

I'm testing different models and my general expectation is that models that have a high coefficient of determination should roughly also have a lower error rate (RMSE in that case) than those with a low R² (please correct me if this premise is wrong).

I've defined three models, linear regression, decision trees and random forests:

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.tree import DecisionTreeRegressor

[some code missing here]

def get_rmse(testdata, reg):
    errorsum = 0
    items = 0
    for index, row in testdata.iterrows():
        y = row["Target"]
        X = np.array(row)[:-1].reshape(1, -1)
        y_hat = reg.predict(X)
        errorsum += (y-y_hat.item())**2
        items += 1
    rmse = np.sqrt(errorsum/items)
    return rmse

reg = LinearRegression().fit(X_train, y_train)
score = reg.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, reg)}")
print("-------")

dt = DecisionTreeRegressor()
dt.fit(X_train, y_train)
score = dt.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, dt)}")
print("-------")

rf = RandomForestRegressor(n_jobs=-1)
rf.fit(X_train, y_train)
score = rf.score(X_test, y_test)
print(f"R²: {score}")
print(f"RMSE: {get_rmse(testdata, rf)}")

The results for my dataset are:

R²: 0.8322231990679154
RMSE: 1.6443917859748052
-------
R²: 0.967768714719696
RMSE: 3.0779040219133758
-------
R²: 0.9772274437532209
RMSE: 2.020003916126851

So it looks like the linear regression model's RMSE is very low but it also has a low R². Why? Shouldn't we expect that a model that makes less mistakes also explains the data better?

Answer 1

sklearn defines score to be the $R^2$:

$$R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}$$

Where $\text{RSS} = n\text{MSE}$ and $\text{TSS} = (n-1)\operatorname{Var}(y)$, the sample variance of $y$.

The RMSE on the other hand is:

$$\text{RMSE} = \sqrt{\text{MSE}}$$

So we can relate one to the other:

$$R^2 = 1 - \frac{\text{RMSE}^2}{\text{TSS}}$$

And

$$\text{RMSE} = \sqrt{(1-R^2)\text{TSS}}$$

So, if $\text{RMSE}$ increases, $R^2$ decreases and vice versa.