# Comparing 2 models with very different R2 values but with very close RMSE values

In my prediction model, there are 20 independent variables, and the dependent variable [y] has a range of 0-4. In the first approach I use LASSO to make a prediction [I have separate training and test sets]. Here is the result of the model evaluation:

METHOD      MAE    R2     RMSE
LASSO       0.561  0.141  0.985


In the second approach, first I train a classification model to identify records whose y value is 0 [y0] and whose y value is non-zero [ynon0]. After the classification, I run the LASSO regression model to predict the ynon0. At the end, I combine the items classified as y0 with the predicted ynon0 to evaluate the performance of ensembling two methods. Here are the results:

METHOD      MAE    R2     RMSE
CLASS+LASSO 0.536  0.437  0.971


I see that the second model explains much more variability in the outcome variable (given the R2 values) whereas the MAE and RMSE has improved slightly. Based on these results, I wonder if I should pay attention to R2, and should still consider the ensembling approach more effective, if my goal is to make an accurate prediction. Or, more generally, in what situation, I am supposed to favour the second approach or the first approach? Any ideas?

• I don't think you can have higher $R^2$ and lower RMSE at the same time, in sample. There must be something fishy going on in the calculations. But if you mean out of sample, then $R^2$ is pretty useless in general as it totally neglects forecast bias and only reflects forecast variance. – Richard Hardy Mar 30 '17 at 11:34
• Sounds like the second approach, apart form being better on all metrics, is also a better model for the data! (sounds like there's some zero-inflation going on there?) – user3903581 Apr 3 '17 at 2:36

Both $R^2$ and RMSE are calculated from the sum of squared errors (SSE), but they are scaled differently. $R^2=1-\frac{SSE}{n \cdot Var[y]}$ and $RMSE=\sqrt{\frac{1}{n}SSE}$.
The effect of this scaling is that if the $R^2$ value is small (like for your first model), then a small improvement in the RMSE results in a large improvement in $R^2$. For instance, if $R^2$ was 0.1, and the RMSE decreased by 30%, then the R^2 value would improve to 0.55!
• I generally use RMSE, and take the ratio to the RMSE of a simple baseline model (such as predicting the mean value). This gives a reasonable notion of relative improvement. The only advantage to $R^2$ is that it has a clearer scaling. – AaronDefazio Apr 3 '17 at 12:46