I am trying to create a linear regression model. I split my data into training and testing data, and built a model. The $R^2$ value on the training data is 0.840. Then I ran the model on the test data. When I calculate the $R^2$, I get 0.982:

y.predicted <- predict(lm1, newdata=test)
y.actual <- donation_test$yval
errors <- (y.actual - y.predicted)
1 - sum(errors^2)/sum(y.actual^2)
[1] 0.9823576  

What I am doing wrong? It seems very unlikely that my model fits my test data better than my training data.

  • 4
    $\begingroup$ I think the correct formula for R-squared should be 1 - sum(errors^2)/sum((y.actual-mean(y.actual))^2) $\endgroup$ – Panos Feb 12 '14 at 9:56
  • 5
    $\begingroup$ This is totally possible depending on how you split your train/test set. I have very similar experience with other ML models, particularly if your data is small and the split is non-random. $\endgroup$ – horaceT Jun 21 '16 at 21:35
  • $\begingroup$ You should always do cross validation on the training data. It gives you confidence and a very good idea of how your model will perform on the unseen data. May be you should do 10 fold cross validation and see the Rsquared value which would be the average of 10 Rsquared values. You may use caret package to do this easily. In your example, you may expect Rsquared value from 10 fold CV to fall between 0.84 - 0.98 and is more closer to 0.98. $\endgroup$ – Rajesh Gupta May 4 '17 at 16:14

I think the formula to calculate r-squared is

R-squared = 1 - (RSS/TSS)

where TSS = sum((y-mean(y))^2) and RSS = sum((y-y.predict)^2)

  • $\begingroup$ Can you provide it as code and test it as well? $\endgroup$ – Joe_74 Jun 21 '16 at 20:54
  • $\begingroup$ Have a look at our editing help to see how to do Latex markup for math typesetting. For instance writing $x$ produces $x$, and $\frac{a}{b}$ produces $\frac{a}{b}$ $\endgroup$ – Silverfish Jun 21 '16 at 21:10
  • $\begingroup$ Alternatively you can use indentation to write a code sample, and surround text with ` on each side to write some code inline. $\endgroup$ – Silverfish Jun 21 '16 at 21:13

$R^2$ value is not a metric for model selection or model fit. The reason for this is that there is inherent variability of data may affect the $R^2$. Consider the following data sets: enter image description here

The (Y_ v/s X) plot has more spread than (Y v/s X). As a result the $R^2$ value for the previous(which has more variance) will be lower than the latter(which has less variance).

This proves that you must not use the $R^2$ value to check if the model is fitting the data well or not.

Instead you should check the model assumptions of :

  • Linear Trend : (from scatter plot)
  • Constant variance of error : (from residual v/s fit plot)
  • Normal distribution of error : (from QQ plot)
  • 1
    $\begingroup$ You provided very little information. The 2 sample sizes are very important to know. Unless they both exceed around 10,000 split sample validation is too noisy to be trusted. The noise (sampling variability) would easily explain why R^2 would not be reliably estimated in the test sample (in addition to checking your formulas). $\endgroup$ – Frank Harrell Dec 27 '17 at 14:44
  • 1
    $\begingroup$ There is a chance this might have occoured. Additionally, plot the test and train dataset and see if the variance of the data is similar in both the cases. If the test dataset variance is lesser than train dataset variance, which is similar to what I explained. $\endgroup$ – user3808268 Dec 27 '17 at 15:20

One explanation might relate to how you subset your test data (they way you split training and testing data). If your test data only consists of (just a few) similar observations then it is very likely for your R-squared measure to be different than that of the training data.

A good practice is to split X% of the data selected randomly into the training set, and the remaining (100 - X)% into your test data.

Also, generally speaking, you should not be using R-squared for your test data but something like RMSE or MSE instead.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.