# Confidence Intervals for Holdout R^2?

Let's say I'm performing regularized regression and I want to validate the results using holdout. (I'm choosing holdout instead of cross-validation because my dataset is fairly large, so computational power is an issue and the difference between holdout and cross-validation estimates will likely be negligible.) I want to put a confidence interval on the $R^2$ score for the holdout set. Is the following procedure valid?

1. Compute the regression coefficients on the training data.

2. Compute the predictions on the held out data. Treat each prediction as if it were an observed random variable.

3. Compute the Pearson correlation of the predictions with the actual $Y$ values for the held-out data. Compute a confidence interval for this correlation using Fisher's Z-Transformation.

4. If the confidence interval of this correlation is some set of real numbers, then the confidence interval of $R^2$ is just the set of the squares of these numbers.

I can't think of any reason why this procedure would be wrong, but I've never heard of it being done before, so I wanted to run it by some more experienced statisticians.

• Why not utilize an approximate standard error for the R-squared value on the hold out. For example using CI.Rsq in the psychometric package? Mar 19, 2011 at 3:11

• Suppose $P(-a < X < b)=p$ and $P(a<X<b)=q$ where $a,b>0$. Then $P(a^2 < X^2 < b^2)=q\ne p$. Mar 20, 2011 at 5:50