3
$\begingroup$

Let's say I'm performing regularized regression and I want to validate the results using holdout. (I'm choosing holdout instead of cross-validation because my dataset is fairly large, so computational power is an issue and the difference between holdout and cross-validation estimates will likely be negligible.) I want to put a confidence interval on the $R^2$ score for the holdout set. Is the following procedure valid?

  1. Compute the regression coefficients on the training data.

  2. Compute the predictions on the held out data. Treat each prediction as if it were an observed random variable.

  3. Compute the Pearson correlation of the predictions with the actual $Y$ values for the held-out data. Compute a confidence interval for this correlation using Fisher's Z-Transformation.

  4. If the confidence interval of this correlation is some set of real numbers, then the confidence interval of $R^2$ is just the set of the squares of these numbers.

I can't think of any reason why this procedure would be wrong, but I've never heard of it being done before, so I wanted to run it by some more experienced statisticians.

$\endgroup$
1
  • $\begingroup$ Why not utilize an approximate standard error for the R-squared value on the hold out. For example using CI.Rsq in the psychometric package? $\endgroup$
    – B_Miner
    Mar 19, 2011 at 3:11

1 Answer 1

2
$\begingroup$

That should work ok provided the lower limit of your confidence interval is positive. Otherwise the square transformation is not monotonically increasing and then the probability coverage is not preserved.

I've never seen it done before either.

$\endgroup$
4
  • 1
    $\begingroup$ Can you explain the part about probability coverage? $\endgroup$
    – B_Miner
    Mar 19, 2011 at 2:39
  • $\begingroup$ Suppose $P(-a < X < b)=p$ and $P(a<X<b)=q$ where $a,b>0$. Then $P(a^2 < X^2 < b^2)=q\ne p$. $\endgroup$ Mar 20, 2011 at 5:50
  • $\begingroup$ Definitely true but not what I was suggesting. Of course if the lower and upper bound of your confidence interval on Pearson correlation have different signs then your lower bound on the confidence interval for R^2 is zero. $\endgroup$
    – dsimcha
    Mar 21, 2011 at 14:50
  • $\begingroup$ @dsimcha. No. If you set the lower bound to zero you will change the probability coverage of the CI. $\endgroup$ Mar 21, 2011 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.