I have a regression model with $R^2$ value. I am trying to use this model on a different data set to see how well the model predicts this data.

So far I have used the model with the 'new' data to make predictions and then I have used a fitted line plot to asses how well the new predictions compare with the real data.

I would like a way of including the original $R^2$ value in this new analysis.

E.g one model has r sq 95.5%, when I plot the predicted values (using this model) against the real data the r sq is 74%.

Is there a better way I can do this analysis rather than just reporting the new $R^2$ value?

  • $\begingroup$ What do you mean by "including the original rsq value in this new analysis"? $\endgroup$ – mkt - Reinstate Monica Aug 2 '19 at 11:38
  • $\begingroup$ So my original model is very good at explaining the variability in the data set that was used to create the model (r sq 95%). But I have now used that model on a new data set and I'm just trying to understand how I can account for the 5% of the variability my model cant explain in these new results. $\endgroup$ – user252775 Aug 2 '19 at 11:51
  • 1
    $\begingroup$ If your model cannot explain that 5%, I'm puzzled by the idea of accounting for it. $\endgroup$ – mkt - Reinstate Monica Aug 2 '19 at 11:53
  • $\begingroup$ Many sources of error cannot be accounted for by a model, such as: transistor amplifier noise, manufacturing variability of equipment, manual recording and transcription errors, and so forth. $\endgroup$ – James Phillips Aug 2 '19 at 12:15
  • $\begingroup$ If you haven't already, you'll want to look into the terms shrinkage and the bias-variance tradeoff. $\endgroup$ – rolando2 Aug 3 '19 at 0:54

If I understand correctly, your objective is to test the out-of-sample performance of a regression model estimated in a certain training set. And you are looking at the decay of the R2. Since the R2 is computed from the variance of residuals, I think it’s the correct way.. just to give you alternative ideas, if your residuals are close to a normal distribution, you could also use the analogy between MLE and OLS to gauge your model based on some metrics that are typically used in MLE contexts, but why over-complicating? I think it’s fine. The simpler, the clearer, the better.

  • 1
    $\begingroup$ Yes, thanks I am looking at the decay of the rsq value and that a really nice way to put it :) thanks again for your help! $\endgroup$ – user252775 Aug 2 '19 at 13:24
  • $\begingroup$ You’re welcome! $\endgroup$ – Fr1 Aug 2 '19 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.