1
$\begingroup$

I am trying to understand in what sense, $R^2$ and $R_{adj}^2$ represent the "explained variance." I can't find any similar question that explores the connection in mathematical detail.

My current understanding is that "explained variance" is shorthand for the ratio of the variance of an estimator (e.g. the regression line) to the variance of the sample data. I've tried to lay out my thinking below but I get a contradiction and I don't understand which assumption I'm making is wrong.

Can anyone help me understand the mistake? I would appreciate any insight!


The standard regression model (not necessarily linear) is something like this:

$$y=\hat{y}+\epsilon$$

where $\hat{y}$ is an unbiased estimator of $y$. The variance of $y$ is:

$$V(y)=V(\hat{y}+\epsilon)=V(\hat{y})+V(\epsilon)+2\text{cov}(\hat{y},\epsilon)$$

Assuming the estimator and error are uncorrelated, the covariance term is zero:

$$V(y)=V(\hat{y})+V(\epsilon)$$

Rearranging in terms of the ratio I have in mind:

$$\frac{V(\hat{y})}{V(y)}=1-\frac{V(\epsilon)}{V(y)}$$

For discrete data, each variance term can be written in terms of the sum of squared errors, regressors, or totals:

$$V(\epsilon)=\frac{1}{DF_\epsilon}\sum (y_i-\hat{y}_i)^2=\frac{\text{SSE}}{DF_\epsilon}$$

$$V(y)=\frac{1}{DF_y}\sum (y_i-\bar{y}_i)^2=\frac{\text{SST}}{DF_y}$$

$$V(\hat{y})=\frac{1}{DF_\hat{y}}\sum (\hat{y}_i-\bar{\hat{y}}_i)^2=\frac{\text{SSR}}{DF_\hat{y}}$$

With $\bar{y}_i=\bar{\hat{y}}_i$. Substituting these definitions into the variance equation yields:

$$\frac{DF_y}{DF_\hat{y}}\frac{\text{SSR}}{\text{SST}}=1-\frac{DF_y}{DF_\epsilon}\frac{\text{SSE}}{\text{SST}}$$

The degrees of freedom are related by:

$$DF_y=DF_\hat{y}+DF_\epsilon \rightarrow (n-1)=(k)+(n-k-1)$$

where n is the number of observations and k is the number of parameters. So the variance equation becomes:

$$\frac{n-1}{k}\frac{\text{SSR}}{\text{SST}}=1-\frac{n-1}{n-k-1}\frac{\text{SSE}}{\text{SST}}$$

where the RHS is the standard definition of ${R_{adj}^2}$.

If the variables ($x$'s) of the estimator $\hat{y}$ are all orthogonal and $\sum y =\sum \hat{y}$ (e.g. by minimizing SSR with respect to a constant parameter in $\hat{y}$) then:

$$\text{SST}=\text{SSR}+\text{SSE}$$

Plugging this into the above yields:

$$\frac{n-1}{k}\left(1-\frac{\text{SSE}}{\text{SST}}\right)=1-\frac{n-1}{n-k-1}\frac{\text{SSE}}{\text{SST}}$$

But the LHS and RHS are clearly not equivalent expressions. Shouldn't this work even in an OLS case? Where is the mistake? Is only the RHS a valid expression? Does $R^2$ "explain the variance" only when k=0?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.