6
$\begingroup$

If a regression has 100 observations and 100 variables and none of them are collinear, what is the r squared? I know that it means that the full rank assumption is satisfied, so does this mean the r squared equals 1? Also, what would the t-statistic for the 5th coefficient be?

$\endgroup$
0

2 Answers 2

15
$\begingroup$

(Most of this a linear algebra question is disguise!)

If the $100\times100$ matrix $X$ is full-rank, that means the columns form a basis for $\mathbb R^{100}$. Since $y\in\mathbb R^{100}$, $y$ can be written as some linear combination of any basis for $\mathbb R^{100}$, such as the set of columns of $X$.

That is, the columns of $X$ perfectly predict $y$, and there is no prediction error (at least not in-sample). Consequently, $y=\hat y$, and $R^2=1$.

$$ R^2=1-\dfrac{ \overset{n}{\underset{i=1}{\sum}}\left(y_i-\hat y_i\right)^2 }{ \overset{n}{\underset{i=1}{\sum}}\left(y_i-\bar y\right)^2 }\\ =1-\dfrac{ \left. \overset{n}{\underset{i=1}{\sum}}\left(y_i-\hat y_i\right)^2 \middle/ n \right. }{ \left. \overset{n}{\underset{i=1}{\sum}}\left(y_i-\bar y\right)^2 \middle/ n \right. } \\ 1-\dfrac{ \text{var}(y-\hat y) }{ \text{var}(y) }\\ =1-\dfrac{0}{\text{var}(y)}=1 $$

(This assumes not all values of $y$ are equal, but if they are, that is not an interesting regression problem.)

With zero residual variance, it does not make much sense to talk about the t-stats for any of the coefficients, since t-stats divide by residual variance and dividing by zero is frowned upon.

$\endgroup$
1
  • 2
    $\begingroup$ $\text{Var}$ (or $\text{var}$) could be a good alternative to $var$. $\endgroup$ Dec 2, 2022 at 16:17
1
$\begingroup$

If none of the explanatory variables is colinear (i.e., each pair has zero correlation) then the coefficient-of-determination for the regresion is equal to the sum of the coefficients-of-determination for individual simple linear regessions of each explanatory variable against the response variable. If you would like to learn more about the relationships between collinearity and the coefficient-of-determination in linear regression, as wel as a broader geometric view of regression, you can find some discussion of this topic in my paper O'Neill (2019).

$\endgroup$
4
  • $\begingroup$ This is certainly true, yet I’m having a very difficult time reconciling it with my post (which also is correct…I hope). I think an implication of our two posts is that, if $99$ of the features are uncorrelated with the outcome, then that $100th$ feature must be perfectly correlated with the outcome, which I’m not sure I’d ever known until right now. $\endgroup$
    – Dave
    Dec 2, 2022 at 5:20
  • $\begingroup$ This is true but not useful, because in the present case, as @Dave's answer shows, we can determine $R^2$ without having to consider any of the partial $R^2$ values. $\endgroup$
    – whuber
    Dec 2, 2022 at 14:25
  • $\begingroup$ @Dave Perhaps the piece that will make you feel at home is to think about how strong of a constraint the pairwise non-colinearity is on explanatory variables. $\endgroup$ Dec 2, 2022 at 15:25
  • 2
    $\begingroup$ @whuber: Yes, sorry all, in my haste to answer I didn't notice the piece of information that number of observations is equal to number of variables, so apply the other answer in that case. I'll leave this answer up anyway, in case you're interested in the more general case where number of observations is greater than number of variables. $\endgroup$
    – Ben
    Dec 2, 2022 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.