Regression models with comparable MAE but differing R²

Question

I have trained two regression models on the same dataset. They perform with comparable mean absolut errors $MAE_{1,2} \approx 0.45$, but the coefficient of determination differs significantly with $R^2_1 \approx 0$ (well, that's bad!) in one case and $R^2_2 \approx 0.4$ (mhh...) in the other.

I understand that a better R² means the model explains the encountered variance better. But how to interpret this in the case of equal MAE? Model two just has a smaller variance?

NB: I understand that my choice of metrics might be questionable.

• I chose R² instead of adjusted R² simply because model two is a deep neural network and I did not find any literature in how to compute it in this case.
• I chose MAE over mean squared error MSE because the values I predict fall into a well-define range $[0,10]$, which makes MAE intuitively meaningful to me, whereas MSE would be not.

If the behavior I observe is due to this choice of metrics I would be glad to learn how this is the case! Also please excuse if this turns out a noob question, my understanding of statistics is somewhat empirical in its nature.

Answer 1

$R^2$ is a function of MSE loss.

$$ R^2=\dfrac{ \sum_i\big( y_i-\hat y_i \big)^2 }{\sum_i\big( y_i-\bar y \big)^2}\\=\dfrac{ nMSE }{\sum_i\big( y_i-\bar y \big)^2} $$

Consequently, if you are comparing two models on their respective $R^2$ values, you are implicitly using MSE. Any model with lower (worse) $R^2$ than another of the same data has a higher (worse) MSE.

Since MAE does not square residuals, large misses are not brutally punished the way they are by MSE.

My interpretation of your results than both models have comparable MAE but different MSE is that one (low $R^2$) tends to make small mistakes with the occasional terrible prediction, while the other tends to make larger mistakes but fewer of the colossal errors.

This is likely to be reflected in plots of your distributions of model residuals.