10
$\begingroup$

This question is a follow-up to a prior question.


Basically, I wanted to study under what conditions when we regress the residuals to $x_1$, we will get $\small R^2$ of 20%.

As a first step to attack this problem, my question is, how do I express $\small R^2$ in matrix form?

Then I will try to express "$\small R^2$ of regressing residuals to $x_1$" using matrix form.

Also, how can I add regression weights into the expression?

$\endgroup$
5
  • 1
    $\begingroup$ Re: Basically I wanted to study under what conditions when we regress the residuals to x1, we will get a R-squared of 20%, if the regression is an ordinary least squares regression, and $x_1$ was included in the model, then the answer is never, and I showed you in my answer here. If you're actually asking something else then please clarify. $\endgroup$
    – Macro
    Jul 27, 2012 at 15:36
  • $\begingroup$ Hi Macro, because I have weights in the regression. I wanted to be able to derive something show study the R^2. That's the reason for asking for the matrix form expression. Thank you! $\endgroup$
    – Luna
    Jul 27, 2012 at 19:06
  • $\begingroup$ Hi @Luna - OK but there is no mention of weights in this post. What kind of weights? Can you edit the post to clarify? $\endgroup$
    – Macro
    Jul 27, 2012 at 19:10
  • $\begingroup$ I had mentioned weights in that thread... here I am looking for a generic form - once I learn the generic form, I can add weights myself... right? $\endgroup$
    – Luna
    Jul 27, 2012 at 21:29
  • $\begingroup$ stats.stackexchange.com/q/351200/119261 $\endgroup$ Jan 9, 2020 at 15:51

2 Answers 2

16
$\begingroup$

We have $$\begin{align*} R^2 = 1 - \frac{\sum{e_i^2}}{\sum{(y_i - \bar{y})^2}} = 1 - \frac{e^\prime e}{\tilde{y}^\prime\tilde{y}}, \end{align*}$$ where $\tilde{y}$ is a vector $y$ demeaned.

Recall that $\hat{\beta} = (X^\prime X)^{-1} X^\prime y$, implying that $e= y - X\hat{\beta} = y - X(X^\prime X)^{-1}X^\prime y$. Regression on a vector of 1s, written as $l$, gives the mean of $y$ as the predicted value and residuals from that model produce demeaned $y$ values; $\tilde{y} = y - \bar{y} = y - l(l^\prime l)^{-1}l^\prime y$.

Let $H = X(X^\prime X)^{-1}X^\prime$ and let $M = l(l^\prime l)^{-1}l^\prime$, where $l$ is a vector of 1's. Also, let $I$ be an identity matrix of the requisite size. Then we have

$$\begin{align*} R^2 &= 1- \frac{e^\prime e}{\tilde{y}^\prime\tilde{y}} \\ &= 1 - \frac{y^\prime(I - H)^\prime(I-H)y}{y^\prime (I - M)^\prime(I-M)y} \\ &= 1 - \frac{y^\prime(I-H)y}{y^\prime (I-M)y}, \end{align*}$$

where the second line comes from the fact that $H$ and $M$ (and $I$) are idempotent.

In the weighted case, let $\Omega$ be the weighting matrix used in the OLS objective function, $e^\prime \Omega e$. Additionally, let $H_w = X \Omega^{1/2} (X^\prime \Omega X)^{-1} \Omega^{1/2} X^\prime$ and $M_w = l \Omega^{1/2}(l^\prime \Omega l)^{-1} \Omega^{1/2} l^\prime$. Then, $$\begin{align*} R^2 &= 1 - \frac{y^\prime \Omega^{1/2} (I-H_w) \Omega^{1/2} y}{y^\prime \Omega^{1/2}(I-M_w) \Omega^{1/2}y}, \end{align*}$$

$\endgroup$
6
  • $\begingroup$ +1 It's nice (and elegant) to see the variance in the denominator pop out from regression against a constant. $\endgroup$
    – whuber
    Jul 27, 2012 at 20:08
  • $\begingroup$ thanks a lot! I upvoted for you. Also, how to add regression weights to the expression? Thank you! $\endgroup$
    – Luna
    Jul 27, 2012 at 21:42
  • 1
    $\begingroup$ @Luna, if this post answers your question & provides the info you need, you should consider accepting it (by clicking the check mark to its left) as well as upvoting it. $\endgroup$ Aug 11, 2012 at 18:13
  • 1
    $\begingroup$ @Charlie, I think you may have accidentally dropped $y$ from your first equation for $\hat\beta$. Also, I don't quite follow your equation for $R^2$, I'm used to seeing $\sum(\hat y_i-\bar y)^2/\sum(y_i -\bar y)^2$ or $1-(\sum(y_i-\hat y_i)^2/\sum(y_i-\bar y)^2)$. I'm interpreting your $e^2$ as $\sum(y_i-\hat y_i)^2$, so I'm confused; is there a way that can make this clearer for me? $\endgroup$ Aug 12, 2012 at 3:45
  • $\begingroup$ @gung, You're right, I had $R^2$ defined incorrectly. I hope that it is correct now. Thanks! $\endgroup$
    – Charlie
    Aug 13, 2012 at 17:59
2
$\begingroup$

You can write the coefficient-of-determination as a simple quadratic form of the correlation values between the individual variables (see this answer for details). Consider a multiple linear regression with $m$ explanatory vectors and an intercept term. Let $r_i = \mathbb{Corr}(\mathbf{y},\mathbf{x}_i)$ and $r_{i,j} = \mathbb{Corr}(\mathbf{x}_i,\mathbf{x}_j)$ and define:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} r_1 \\ r_2 \\ \vdots \\ r_m \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} r_{1,1} & r_{1,2} & \cdots & r_{1,m} \\ r_{2,1} & r_{2,2} & \cdots & r_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ r_{m,1} & r_{m,2} & \cdots & r_{m,m} \\ \end{bmatrix}.$$

With a bit of linear algebra it can be shown that:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}}.$$

The square-root of the coefficient-of-determination gives the multiple correlation coefficient, which is a multivariate extension of the absolute correlation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.