In my prediction model, there are 20 independent variables, and the dependent variable [y] has a range of 0-4. In the first approach I use LASSO to make a prediction [I have separate training and test sets]. Here is the result of the model evaluation:

METHOD      MAE    R2     RMSE
LASSO       0.561  0.141  0.985

In the second approach, first I train a classification model to identify records whose y value is 0 [y0] and whose y value is non-zero [ynon0]. After the classification, I run the LASSO regression model to predict the ynon0. At the end, I combine the items classified as y0 with the predicted ynon0 to evaluate the performance of ensembling two methods. Here are the results:

METHOD      MAE    R2     RMSE
CLASS+LASSO 0.536  0.437  0.971

I see that the second model explains much more variability in the outcome variable (given the R2 values) whereas the MAE and RMSE has improved slightly. Based on these results, I wonder if I should pay attention to R2, and should still consider the ensembling approach more effective, if my goal is to make an accurate prediction. Or, more generally, in what situation, I am supposed to favour the second approach or the first approach? Any ideas?

  • 1
    $\begingroup$ I don't think you can have higher $R^2$ and lower RMSE at the same time, in sample. There must be something fishy going on in the calculations. But if you mean out of sample, then $R^2$ is pretty useless in general as it totally neglects forecast bias and only reflects forecast variance. $\endgroup$ – Richard Hardy Mar 30 '17 at 11:34
  • $\begingroup$ Sounds like the second approach, apart form being better on all metrics, is also a better model for the data! (sounds like there's some zero-inflation going on there?) $\endgroup$ – user3903581 Apr 3 '17 at 2:36

Both $R^2$ and RMSE are calculated from the sum of squared errors (SSE), but they are scaled differently. $R^2=1-\frac{SSE}{n \cdot Var[y]}$ and $RMSE=\sqrt{\frac{1}{n}SSE}$.

The effect of this scaling is that if the $R^2$ value is small (like for your first model), then a small improvement in the RMSE results in a large improvement in $R^2$. For instance, if $R^2$ was 0.1, and the RMSE decreased by 30%, then the R^2 value would improve to 0.55!

I'm a little confused about your discussion about which approach is better. The second approach (the more complex model) is better by all three metrics. Recall that for RMSE and MAE lower is better.

  • $\begingroup$ Thanks for the answer! What I meant was, if I consider that the improvement was very minor in RMSE (in a given context), then why would not follow the simple model which is easier to train? Whereas, if I consider that the improvement was large in R2, I would definitely prefer the more complex model. But, I guess if my focus is on the prediction accuracy, RMSE and MAE are better metrics to use in my case. What do you think? $\endgroup$ – renakre Apr 3 '17 at 10:17
  • $\begingroup$ I generally use RMSE, and take the ratio to the RMSE of a simple baseline model (such as predicting the mean value). This gives a reasonable notion of relative improvement. The only advantage to $R^2$ is that it has a clearer scaling. $\endgroup$ – AaronDefazio Apr 3 '17 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.