5
$\begingroup$

I have a dataset that I divide into two equal partitions A and B.

I estimate a regression model on partition A.

I want to calculate the cross-validated $R^2$ when predicting the values in partition B.

I would like to know if the following approach is correct and also what other ways there could be:

#generate data:

data <- replicate(10, rnorm(100))
data <- as.data.frame(data)

#divide into training and test set:

train <- data[1:50,]
test <- data[51:100,]

#fit model and get predictions for unseen data:

model <- lm(train[,1] ~., data = train)
predictions <- predict(model, test)

#obtain cross-validated R squared:

cor(predictions,test[,1])^2
$\endgroup$
6
$\begingroup$

That is incorrect because it allows for recalibration of predictions with a new overall slope and intercept. Use this formula after freezing all coefficients: 1 - (sum of squared errors) / (sum of squares total). The denominator is $(n-1)\times$ the observed variance of $Y$ in the holdout sample. When you do it correctly you can get negative $R^2$ in some holdout samples when the real $R^2$ is low.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your help! So the correct formula is 1-(sum((data[,1] - predictions)^2) / (n-1) * var(test[,1]) ? $\endgroup$ – user3742038 Dec 12 '15 at 16:21
  • $\begingroup$ p.s. what do you mean by freezing all coefficients? $\endgroup$ – user3742038 Dec 12 '15 at 16:22
  • 1
    $\begingroup$ (+1) "Freezing" means you don't get to refit your model on the test set. The test set is a proxy for any additional data to which you would like to apply the model you have--which at this stage is the one fit only on the training data. (I have recently discovered, by observing what my own students do, that the misconception reflected in this question is a common one. I found that displaying a plot of predicted vs. actual values for the test set and discussing its construction could alleviate this confusion.) $\endgroup$ – whuber Dec 12 '15 at 16:26
  • $\begingroup$ @whuber: I don't get it. Why does my approach allow for recalibration of predictions? I am estimating the model on one partition and using it to predict unseen data. Can you please clarify which part of my approach is incorrect? Thank you! $\endgroup$ – user3742038 Dec 12 '15 at 17:10
  • 2
    $\begingroup$ It is very incorrect. Say your predictions are off by a multiple of 2 and an additional increment of 10. The correlation coefficient will not penalize for that, internally allowing for a new slope and intercept to be estimated on top of the predictions. A correlation is a relative measure whereas sum of squared errors is absolute. $\endgroup$ – Frank Harrell Dec 13 '15 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.