1
$\begingroup$

I have a dataset where the values of the dependent variable are small, i.e. range is 1.7 to 2.2. I fitted a model and made predictions on y_test, the predictions are good because the MAE is 0.3. All the predictions are almost close to y_test.

When I'm calculating r2_score, the r2 value is -16, as the data points are small and their range is small. The sum of squares of the residual errors is far greater than the total sum of the errors. That is the reason why r2_score is very low. Refer the dataframe below. dataframe R2 score

Did you see the above pics, the SSres is very huge where as SStot is very small. The difference between true values and predictions is on average 0.3, still why R2 score is so low? How to overcome this problem? Should I ignore this?

$\endgroup$
3
  • $\begingroup$ R-squared should generally be between 0 and 1, where 0 is chance. Less than zero is possible, but indicates a very poor model, or a big difference between training and test value. A value of -16 means something is incredibly wrong in either your data, your model, or your code. $\endgroup$
    – Eoin
    Feb 17, 2022 at 11:41
  • $\begingroup$ Is an MAE of 0.3 good when the range of values is so small? If a model simply assigned 1.95 to every sample then the worst-case MAE would be 0.25 $\endgroup$
    – A. Bollans
    Jul 16, 2022 at 9:24
  • $\begingroup$ A NIST 4-plot on your residual should help you with your diagnosis. $\endgroup$ Jul 16, 2022 at 10:29

1 Answer 1

2
$\begingroup$

(Let me start by saying that a likely answer is that there is a bug in your code.)

Consider what $R^2$ means: it is a comparison of your model to the performance of a model that naïvely guesses the mean every time.

$$ R^2=1-\dfrac{ \sum\big( y_i-\hat y_i \big)^2 }{\sum\big( y_i-\bar y \big)^2} $$

If your performance is worse than the baseline model whose predictions are used in the denominator, then you wind up with a value less than zero, indicating that you get stronger performance just by guessing $\bar y$ every time.

If your value is $-16$, this tells me that you’re able to do much better just by guessing $\bar y$ every time.

I can think of a couple of reasons why you might have terrible $R^2$ yet acceptable MAE.

  1. There are a few (maybe just one) terrible predictions that the squaring in $R^2$ penalizes very brutally, but MAE does not punish nearly as severely.

  2. A baseline model that always guesses $\bar y$ achieves better (lower) MAE than your model, and the fact that you’re working with a small range of numbers tricks you into thinking that the small MAE is acceptable, despite stronger performance by a naïve model. Consider calculating the following quantity and noting if the value is less than zero.

$$ 1-\dfrac{ \sum\big\vert y_i-\hat y_i \big\vert }{\sum\big\vert y_i-\bar y \big\vert} $$

$\endgroup$
3
  • $\begingroup$ Thank you @Dave, I figured out what went wrong. I used Relu activation function when there are few negative values in target variable, There are around 40k records, lot of them have negative value, as I used Relu activation, all the negative predictions are made into positive, that is why, r2 score is so bad. $\endgroup$ Feb 19, 2022 at 6:16
  • $\begingroup$ @MasterBlasterCoder But then why is your MAE decent? Or is it not? Did you do that calculation I gave at the end? $\endgroup$
    – Dave
    Feb 19, 2022 at 12:01
  • $\begingroup$ I changed the activation function to linear and the R2 score improved to 0.83. Looks like there were many 0.0 predictions when the actual values are in negative, due to which residual error is increased. $\endgroup$ Feb 19, 2022 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.