1
$\begingroup$

Assume we have a vector of ones and some (one or more) variables $X$, and we run a linear regression of unity on $X$ (i.e. if we only assume one RHS variable, the implied scatter plot would be a flat horizontal line). My questions are:

  1. What would be intuitively the purpose of such a regression?
  2. Given that $R^2 (= 1-RSS/TSS = MSS/TSS)$, is the proportion of the dependent variable explained by the model, and since in this setup $TSS=0$, I expected that statistical software would not return the $R^2$ figure in such a regression. However, they do (see *). I have identified two different conventions statistical packages use for calculating $R^2$, in this special regression case. The difference is on how they calculate $TSS$:
    • $TSS = T$ (where $T =$ # of obs)
    • $TSS = 2RSS$ (since $MSS=RSS$, due to $y_i = \bar y$)

The 2nd question is what is the rationale behind each of the two conventions for calculating $R^2$?

*For instance, for a regression of unity on a single $X$ without an intercept, $R^2$ in Stata (command: regress) and Excel (LINEST) is 0.83, while in MATLAB (fitlm) is 0.5. These figures imply that Stata and MS Excel use the first $TSS$ convention, while MATLAB uses the second approach.

$\endgroup$
2
  • 1
    $\begingroup$ To what software do you refer and what value of $R^2$ does it return? $\endgroup$
    – whuber
    Commented May 6, 2020 at 18:43
  • $\begingroup$ @whuber using Stata I get R-squared 0.8330 and Adj R-squared 0.8323. In MATLAB, with the same set of two variables (unity and a single $X$), I get 0.5 for both R-squared and Adj R-squared. $\endgroup$
    – CKara
    Commented May 6, 2020 at 23:01

1 Answer 1

5
$\begingroup$

The short answer is that such a regression is completely meaningless. See this question for a great discussion of it. Basically, since $y$ is always the same value, you don't observe any variation in it, and so intuitively you will always find that the intercept is equal to $y$, and all the other $\beta = 0$. Basically, you conclude that changing $x$ has no effect on $y$. But this is a special case of that, which can happen when $y$ is just independent of $x$, as any variable you used instead of $x$ would yield the exact same result. Furthermore, as the answers in that question reveal, there's really no concept of $R$ or $R^2$ in this case.

Now as to why you may observe an $R^2$ using statistical software, this is most likely because such software numerically solves for regressions, and so you won't exactly get that $\beta = 0$, but rather that $\beta \approx 0$. In particular, using using the example of R's code, the documentation for lm calculates R squared as $$R^2 = \frac{mss}{mss+rss}$$ where $mss = \sum_i(y_{i,fit} - \bar{y})^2$ and $rss = \sum_i(y_{i,fit} - y_i)^2$. But in your example, both of these values are incredibly small (due to approximation) but non-zero and equal as $y = \bar{y}$, and so you have $mss = rss$ and so $$R^2 = \frac{mss}{mss+rss} = 1/2 =.5$$ which is what I suspect many software might conclude.

So what's going on? Well in this case, R used the equality that $mss + rss = tot$, where $tot = \sum_i ((y_i - \bar{y})^2)$. So instead of calculating $tot$ to get the denominator of $R^2$, it instead used $mss + rss$, but in this case, $tot = 0$ but $mss + rss \neq 0$ due to approximation issues, but again fundamentally the concept of $R^2$ doesn't exist here, and so these equations fail to hold.

See below for an example reproducing the problem in R:

n = 1000
y = rep(1,n)
x = rnorm(n, mean = 100)

reg = lm(y~x)

r = reg$residuals
f = reg$fitted.values

mss = sum((f - mean(f))^2)
rss <- sum(r^2)

rsq = mss/(mss+rss)
[1] 0.499924

UPDATE (after comments)

Let's now consider forcing no intercept, and asked in the comments of this post by OP. First, the intuitive purpose of this regression would be to recover identified information of $X$ without needing $Y$, and so the regression would be 'useless' in that sense: all the information would be contained in the data $X$ without $Y$. This should make sense: the observed $Y$ is a constant. On a more fundamental sense, let's look at the model you're considering. Sticking to the simple linear regression case, you're saying that the underlying model is $y = f(x) = \beta x + \epsilon$, and so $E[y|x=0] = 0$, but you told me $E[y|x = 0] = y = 1$ as $y$ is a constant, so your model is already mis-specified.

Anyway, in the simple linear regression case, we have that without intercept, $y = \beta x$ and $$\hat{\beta} = \frac{\bar{xy}}{\bar{x^2}} = \frac{\bar{x}}{\bar{x^2}}$$

where the last equality follows because $y = 1$ for all observations. You also asked that $\bar{x} = 0$, so $\hat{\beta} = 0$ in your example regardless, unless $\bar{x^2} = 0$, in which case it's undefined. So you know exactly what $\hat{\beta}$ is, and what it's identifying, and so the interpretation of a regression is equivalent to just studying $X$ on it's own. As for why various statistical software gives various $R^2$, I'd check documentation and go from there knowing what you identify.

I'd also be remiss not to link this post on the pitfalls of $R^2$.. at least when I first learned about $R^{2}$, I certainly gave it way more attention than I should have, thinking it's somehow the 'ideal' measure of a regression.

$\endgroup$
8
  • $\begingroup$ In your answer, and the post you suggested, both assume that there is an intercept. In that case, not only $TSS$, but also $RSS$ is 0 (since for $Y=\beta_0+\beta_1X +e$, we'll have $\hat\beta_0=1$ and $\hat\beta_1=0$). If we consider a model without an intercept, then even though $TSS$ is still 0, $RSS$ is not (since for $Y=\beta_1X +e$, the $\hat\beta_1 \neq 0$). So, the model with intercept indeed fits the data perfectly (hence $RSS=0$), but the model without the intercept, does not. The only commonality I see between the two is that in both cases, $TSS=0$, hence the variance of $Y$ is 0. $\endgroup$
    – CKara
    Commented May 7, 2020 at 13:45
  • $\begingroup$ So, in the case with the intercept, indeed the regression might be meaningless. However, I am not convinced that the same is true for the model without the intercept. This latter case of the model without an intercept, and when $R^2$ is calculated using the convention of $TSS=T$, is used as a test statistic (defined as $T*R^2$) by Giacomini-White (2006) in evaluating relative accuracy of competing forecasting models. Hence my question for the meaningfulness of the regression. $\endgroup$
    – CKara
    Commented May 7, 2020 at 13:48
  • $\begingroup$ You pointed out that $RSS=MSS$. Which implies that $TSS=2RSS$. Therefore, that justifies the rationale behind MATLAB’s computation of $R^2$. However, what is the rationale for computing $R^2$ by setting $TSS=T$ (such as in Stata and Excel)? $\endgroup$
    – CKara
    Commented May 7, 2020 at 13:51
  • $\begingroup$ @CKara I updated my post. $\endgroup$
    – doubled
    Commented May 7, 2020 at 16:29
  • $\begingroup$ I have already updated the post to include the two different ways that statistical packages calculate $R^2$. In the case of Stata they set $TSS=T$. In MATLAB they do what you have described above i.e. use the fact that $RSS=MSS$, implying that $TSS=2RSS$. The question remaining to be answered is what is the logic behind setting $RSS=T$, and calculating the goodness of fit as $R^2=1-RSS/T=1-\bar e$? $\endgroup$
    – CKara
    Commented May 7, 2020 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.