I am interested in the distribution of/performing inference on the 'achieved' $R^2$ coefficient in multiple linear regression. Suppose that $y\sim x\beta + \epsilon$ with $\epsilon \sim \mathcal{N}\left(0,\sigma^2\right)$, where $x$ is a $p$-vector. You observe $x_1,x_2,\ldots,x_n$ and corresponding $y_1,y_2,\ldots,y_n$, with independent errors. You perform linear regression to get $\hat{\beta}$, and then compute the sample $R^2$ in the usual way: $R^2 = 1 - \frac{SS_\mbox{err}}{SS_{\mbox{tot}}}$. Nothing new here.

There are known methods for hypothesis testing the population analogue of $R^2$, as well as computing confidence intervals on it. (I am thinking of Lee's paper, and followups by Algina, O'Brien, inter alios). The population analogue is defined in terms of the (unknown) true regression coefficients $\beta$ and the true variance $\sigma^2$.

I am interested in the achieved $R^2$ which is the amount of variance explained by $\hat{\beta}$. Conditional on $x$ and on $\hat{\beta}$, I would define it as $$ R^2_{\mbox{ach}} = 1 - \frac{\mbox{E}\left[\left(y - \hat{\beta}x\right)^2\right]}{\mbox{E}\left[\left(y-\bar{y}\right)^2\right]}. $$ Clearly this is less than the population $R^2$, because the true $\beta$ minimizes $\mbox{E}\left[\left(y - \beta x\right)^2\right]$. I would guess it is less than the sample $R^2$ because that (usually) overestimates the population $R^2$.

I am having problems thinking about this quantity because it is both unobserved and random ($\hat{\beta}$ is a random variable.) I am guesing that, up to some transforms, I can represent this as a non-central chi-square, abusing $\hat{\beta}\sim\mathcal{N}\left(\beta,\sigma^2\left(X^{\top}X\right)^{-1}\right)$, and noting that $\mbox{Var}\left[y - \hat{\beta}x\right] = \sigma^2 + \mbox{Var}\left[(\hat{\beta} - \beta)x\right]$, but I'm being dense and lazy. (Also, treating $x$ as fixed will later be generalized to $x$ being random.)

Is this is a well-known problem? For predicting future performance of a linear model, it would seem to be a more important quantity than the population $R^2$, for example.

  • $\begingroup$ Re "clearly this is less than": this statement appears to be equivalent to asserting that the OLS estimate of $\sigma^2$ is biased! Am I missing something here? $\endgroup$ – whuber Jun 12 '12 at 19:04
  • $\begingroup$ @whuber, hmm. the achieved $R^2$ is computed in terms of the population parameters, so I'm not sure how the OLS estimate of $\sigma^2$ enters into it. $\endgroup$ – shabbychef Jun 12 '12 at 19:19
  • $\begingroup$ ahh, I see the confusion; the $\mbox{Var}$ is conditional on $x$ and $\hat{\beta}$. That is, I have run the regression, computed $\hat{\beta}$ and then will use the regression model 'out of sample'. I can see how this is confusing. edit coming. $\endgroup$ – shabbychef Jun 12 '12 at 19:22
  • 1
    $\begingroup$ the problem is that 'conditional on $x$' forces $y-\hat{\beta}x$ to be just a shift of $y$ and so $R^2_{\mbox{ach}}$ is just identically zero as defined. As noted, I will later have to consider $x$ as being normally distributed, in which case, I believe, the achieved $R^2$ as defined is no longer forced to be zero. bleah. $\endgroup$ – shabbychef Jun 12 '12 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.