4
$\begingroup$

I am using a random forest regression model to make predictions and leave one out cross validation for my prediction task. I am having a difficult time understanding why my R2 score is negative when the MSE, RMSE, and MAE are all very low. Here I am providing a sample of my true and predicted values:

True Value: 0.0511350891441389, Predicted Value: 0.1570743965948912
True Value: 0.1019683613090206, Predicted Value: 0.06101801962025982
True Value: 0.0722484077136202, Predicted Value: 0.12989937556879136
True Value: 0.8151465997429149, Predicted Value: 0.11910986913415476
True Value: 0.0141580461529044, Predicted Value: 0.10300264949635973
True Value: 0.0759365903712855, Predicted Value: 0.2007470535994329
True Value: 0.0168830791575889, Predicted Value: 0.0867039544973983
True Value: 0.0280480358233258, Predicted Value: 0.3334096609357363
True Value: 0.0119374073771543, Predicted Value: 0.0456333839555339
True Value: 0.0879195861169952, Predicted Value: 0.12158770472179008
True Value: 0.1877777777777777, Predicted Value: 0.1636636091524143
True Value: 0.1319864052287581, Predicted Value: 0.05390845919789602

These are the scores:

Mean Squared Error (MSE): 0.035323866926619006
Mean Absolute Error (MAE): 0.1288933724806987
Root Mean Squared Error (RMSE): 0.1879464469646048
R-squared (R2) Score: -0.4162881141285679

I am also providing my visualization of the actual vs. predicted values:

Actual v/s Predicted value plot

$\endgroup$
3
  • $\begingroup$ The reverse situation: high $R^2$ but high MSE $\endgroup$
    – Dave
    Sep 20, 2023 at 18:21
  • 10
    $\begingroup$ Who says that the errors are very low? Your MAE and RMSE are around 0.12 and 0.18, whereas the mean and median of your true values are even smaller. If your MAE is over 100% of the average prediction, that is normally something that I would consider pretty high. $\endgroup$
    – Obie 2.0
    Sep 21, 2023 at 10:32
  • $\begingroup$ @Obie2.0 Yes, thank you for pointing that out. That makes sense to me $\endgroup$
    – Rai
    Sep 21, 2023 at 14:25

2 Answers 2

12
$\begingroup$

If you're getting $R^2<0$, then I assume you're using the equation below, which is the calculation used by sklearn.metrics.r2_score.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) =1-\left(\dfrac{ N\times RMSE^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

This $R^2$ is a function of both the (R)MSE and the total sum of squares. Therefore, no matter how small that (R)MSE is, if the denominator is smaller than the numerator, you will get $R^2<0$.

The interpretation of $R^2<0$ is that your predictions have a higher MSE than that of a naïve model that always predicts the overall mean, $\bar y$. That is, the predictions are not very good. Based on the graph, that appears to be the case, likely driven by that point at the far right that your model misses badly.

$\endgroup$
10
  • 1
    $\begingroup$ Given this insight, it's apparent that my current model might not be suitable for the dataset, especially since it seems highly sensitive to outliers. Considering the characteristics of my dataset, which is relatively small, and the sensitivity to outliers, I'm inclined to believe that I may need a more robust regression model that can handle such scenarios effectively. Could you kindly recommend a regression model or approach that is known for its robustness and is suitable for smaller datasets with potential outliers? Thank you once again for your help. $\endgroup$
    – Rai
    Sep 20, 2023 at 18:13
  • 3
    $\begingroup$ @Rai That's really a new question that warrants its own post, but it's almost asking how to do predictive modeling or machine learning. However, if you only have the $28$ points that I believe I counted, there is little hope of doing any serious predictive modeling. $\endgroup$
    – Dave
    Sep 20, 2023 at 18:14
  • 1
    $\begingroup$ @FrankHarrell What do you mean by "absolute accuracy" as opposed to "accuracy"? $\endgroup$
    – Dave
    Sep 21, 2023 at 12:24
  • 1
    $\begingroup$ Examples: absolute accuracy is a measure related to $Y$ as a function of $\hat{Y}$ (e.g., a smooth calibration curve) and relative accuracy is something like $R^2$ or AUROC (concordance probability). $\endgroup$ Sep 21, 2023 at 18:06
  • 1
    $\begingroup$ Yes if you use the $R^2$ formula that allows $R^2$ to be negative. If you square the Pearson correlation coefficient then it's a relative measure. $\endgroup$ Sep 22, 2023 at 1:39
9
$\begingroup$

I'd like to point out that your RMSE, MSE and MAE are not small at all.

The calibration plot shows only 3 cases (or 10 % of the data) with actual abnormality > 0.2. That is, 90 % of your data points lie within a range of actual abnormality that is just one RMSE wide.
Considering the predictions, 2 RMSE cover almost the complete prediction range.


I'm analytical chemist, an two common heuristics we use may be roughly translated to your application as:

  • qualitative detection is considered barely possible where relative RMSE* is < 1/3 (on first approximation one may argue that this would be fulfilled for the 0.8 actual abnormality point, but prediction is even worse for the high actual abnormality cases, so I'd say, not fulfilled anywhere in your data)

  • Relative RMSE below 10 % is frequently used as minimum requirement for quantitation.

* relative RMSE = $\frac{RMSE}{y}$ or $\frac{RMSE (y)}{y}$; we often look at RMSE as function of the true value.

$\endgroup$
1
  • 6
    $\begingroup$ +1 This is consistent with what $R^2<0$ is showing, that what appear to be small numbers for (R)MSE and MAE aren't all that small for this particular data set. $\endgroup$
    – Dave
    Sep 21, 2023 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.